[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:29 dist-test-slave-6718 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:29 dist-test-slave-6718 krb5kdc[395](info): set up 2 sockets
Sep 28 05:10:29 dist-test-slave-6718 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 05:10:30.979813 388 mini_kdc.cc:121] Time spent starting KDC: real 2.212s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:10:31 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:10:31.154057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43011
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:31.475198 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:31.475495 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:31.475673 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:31.483153 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:31.483256 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:31.483376 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:31.483481 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:31.483558 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:31.493104 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:31.494410 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:31.495983 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:31.508407 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:31.508423 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:31.508374 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:31.509063 408 server_base.cc:1034] running on GCE node
I20260928 05:10:31.509892 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:31.511368 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:31.512637 408 hybrid_clock.cc:648] HybridClock initialized: now 1790572231512544 us; error 75 us; skew 500 ppm
I20260928 05:10:31.512986 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:31 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:31.522094 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:31.523614 408 webserver.cc:460] Webserver started at http://127.0.97.62:40151/ using document root <none> and password file <none>
I20260928 05:10:31.524281 408 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:31.524377 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:31.525084 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:31.528164 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "1be125596e06466385e42bea01b194ae"
format_stamp: "Formatted at 2026-09-28 05:10:31 on dist-test-slave-6718"
server_key: "9157ea856bbdc3fc5cda41bc4f240253"
server_key_iv: "28f984f3d8218ba703e6512deb974560"
server_key_version: "encryptionkey@0"
I20260928 05:10:31.528842 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "1be125596e06466385e42bea01b194ae"
format_stamp: "Formatted at 2026-09-28 05:10:31 on dist-test-slave-6718"
server_key: "9157ea856bbdc3fc5cda41bc4f240253"
server_key_iv: "28f984f3d8218ba703e6512deb974560"
server_key_version: "encryptionkey@0"
I20260928 05:10:31.533902 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:10:31.537362 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:31.538790 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:10:31.538968 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "1be125596e06466385e42bea01b194ae"
format_stamp: "Formatted at 2026-09-28 05:10:31 on dist-test-slave-6718"
server_key: "9157ea856bbdc3fc5cda41bc4f240253"
server_key_iv: "28f984f3d8218ba703e6512deb974560"
server_key_version: "encryptionkey@0"
I20260928 05:10:31.539149 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:31.551393 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:31.556139 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:31.556455 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:31.575736 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43011
I20260928 05:10:31.575780 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43011 every 8 connection(s)
I20260928 05:10:31.577109 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:10:31.580054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 408
I20260928 05:10:31.580363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:10:31.582103 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:31.582883 388 external_mini_cluster.cc:1428] Setting key bb7dc0af4197e9d676f06b96650e2879
I20260928 05:10:31.593720 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae: Bootstrap starting.
I20260928 05:10:31.596675 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:31.598116 475 log.cc:824] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:31.601351 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae: No bootstrap required, opened a new log
I20260928 05:10:31.610518 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be125596e06466385e42bea01b194ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43011 } }
I20260928 05:10:31.610953 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:31.611094 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be125596e06466385e42bea01b194ae, State: Initialized, Role: FOLLOWER
I20260928 05:10:31.611729 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be125596e06466385e42bea01b194ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43011 } }
I20260928 05:10:31.611925 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:10:31.612087 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:10:31.612249 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 0 FOLLOWER]: Advancing to term 1
Sep 28 05:10:31 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:31.614687 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be125596e06466385e42bea01b194ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43011 } }
I20260928 05:10:31.615283 475 leader_election.cc:302] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be125596e06466385e42bea01b194ae; no voters:
I20260928 05:10:31.616466 475 leader_election.cc:288] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:10:31.616591 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:31.618464 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [term 1 LEADER]: Becoming Leader. State: Replica: 1be125596e06466385e42bea01b194ae, State: Running, Role: LEADER
I20260928 05:10:31.618821 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:10:31.619095 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be125596e06466385e42bea01b194ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43011 } }
I20260928 05:10:31.624109 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be125596e06466385e42bea01b194ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be125596e06466385e42bea01b194ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43011 } } }
I20260928 05:10:31.624934 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:31.625442 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be125596e06466385e42bea01b194ae. Latest consensus state: current_term: 1 leader_uuid: "1be125596e06466385e42bea01b194ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be125596e06466385e42bea01b194ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43011 } } }
I20260928 05:10:31.625722 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:31.628844 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:10:31.628873 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:31.587477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.0.97.62:43011)
0928 05:10:31.588494 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:31.588510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:31.589762 (+ 1252us) server_negotiation.cc:407] Connection header received
0928 05:10:31.599334 (+ 9572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:31.599362 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:31.599865 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:31.600311 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:31.603400 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:31.605515 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:31.607355 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:31.608431 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:31.608757 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:31.615188 (+ 6431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:31.615198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:31.615217 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:31.615260 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:31.620009 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:31.621322 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:31.621334 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:31.621345 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:31.621613 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:31.622291 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:31.622299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:31.622305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:31.623214 (+ 909us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:31.626313 (+ 3099us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:31.626778 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 05:10:31.627289 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":424,"threads_started":1}
I20260928 05:10:31.634063 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:10:31.645140 488 catalog_manager.cc:1347] Generated new cluster ID: 836f01eda7564842a1617433c0757604
I20260928 05:10:31.645291 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:10:31.677431 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:10:31.678849 488 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:10:31.688519 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be125596e06466385e42bea01b194ae: Generated new TSK 0
I20260928 05:10:31.689344 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:10:31.741818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43011
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:32.097649 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:32.097924 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:32.098083 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:32.105425 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:32.105533 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:32.105760 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:10:32.119369 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:32.120803 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:32.122195 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:32.136834 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:32.137027 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:32.138154 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:32.139179 502 server_base.cc:1034] running on GCE node
I20260928 05:10:32.139817 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:32.140785 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:32.141994 502 hybrid_clock.cc:648] HybridClock initialized: now 1790572232141946 us; error 36 us; skew 500 ppm
I20260928 05:10:32.142360 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:32 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:32.145740 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:10:32.147302 502 webserver.cc:460] Webserver started at http://127.0.97.1:43493/ using document root <none> and password file <none>
I20260928 05:10:32.148015 502 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:32.148164 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:32.148475 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:32.151528 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "fd05a64687ee466da4db0e49dea2bb36"
format_stamp: "Formatted at 2026-09-28 05:10:32 on dist-test-slave-6718"
server_key: "b72ccbe203816819b2d371d54aab64d7"
server_key_iv: "e4856c2a06321a50cc0c479a5fa815f0"
server_key_version: "encryptionkey@0"
I20260928 05:10:32.152350 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fd05a64687ee466da4db0e49dea2bb36"
format_stamp: "Formatted at 2026-09-28 05:10:32 on dist-test-slave-6718"
server_key: "b72ccbe203816819b2d371d54aab64d7"
server_key_iv: "e4856c2a06321a50cc0c479a5fa815f0"
server_key_version: "encryptionkey@0"
I20260928 05:10:32.157542 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:10:32.161391 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:32.162823 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:10:32.162990 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "fd05a64687ee466da4db0e49dea2bb36"
format_stamp: "Formatted at 2026-09-28 05:10:32 on dist-test-slave-6718"
server_key: "b72ccbe203816819b2d371d54aab64d7"
server_key_iv: "e4856c2a06321a50cc0c479a5fa815f0"
server_key_version: "encryptionkey@0"
I20260928 05:10:32.163192 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:32.188527 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:32.192822 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:32.193110 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:32.194109 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:32.195461 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:32.195561 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:32.195709 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:32.195780 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:32.232194 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41647
I20260928 05:10:32.232247 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41647 every 8 connection(s)
I20260928 05:10:32.233932 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:10:32.242306 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 502
I20260928 05:10:32.242558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:10:32.243278 388 external_mini_cluster.cc:1428] Setting key 9d06e1c829ab423398f95bff60814efd
Sep 28 05:10:32 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:32.258834 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:32.236461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49757 (local address 127.0.97.62:43011)
0928 05:10:32.236943 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:32.236948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:32.237776 (+ 828us) server_negotiation.cc:407] Connection header received
0928 05:10:32.239728 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:32.239733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:32.239823 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:32.239939 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:32.241411 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.242390 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.244426 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.245046 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.245206 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:32.250869 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:32.250878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:32.250885 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:32.250933 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:32.255119 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:32.255967 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:32.255976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:32.255982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:32.256150 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:32.256585 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:32.256592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:32.256598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:32.256985 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:32.257141 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:32.258034 (+ 893us) server_negotiation.cc:299] Negotiation successful
0928 05:10:32.258297 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":282,"threads_started":1}
I20260928 05:10:32.259485 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:32.236927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43011 (local address 127.0.97.1:49757)
0928 05:10:32.237602 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:32.237646 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:32.239443 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:32.240169 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:32.240177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:32.240678 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:32.241242 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:32.241259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.242590 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.242595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:32.244294 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:32.244310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.245387 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.245395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:32.245704 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:32.246442 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:32.246482 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:32.250515 (+ 4033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:32.255317 (+ 4802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:32.255325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:32.255340 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:32.255813 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:32.256261 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:32.256264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:32.256267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:32.256445 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:32.257121 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:32.257127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:32.257517 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 05:10:32.257849 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 05:10:32.258249 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":263,"threads_started":1}
I20260928 05:10:32.261075 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43011
I20260928 05:10:32.261443 630 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:10:32.262327 630 heartbeater.cc:511] Master 127.0.97.62:43011 requested a full tablet report, sending...
I20260928 05:10:32.264892 440 ts_manager.cc:194] Registered new tserver with Master: fd05a64687ee466da4db0e49dea2bb36 (127.0.97.1:41647)
I20260928 05:10:32.266878 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:10:32.291772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43011
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:32.624682 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:32.624956 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:32.625048 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:32.632884 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:32.632992 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:32.633104 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:10:32.643359 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:32.644506 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:32.645777 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:32.658998 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:32.658946 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:32.659029 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:32.659667 636 server_base.cc:1034] running on GCE node
I20260928 05:10:32.660264 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:32.661224 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:32.662415 636 hybrid_clock.cc:648] HybridClock initialized: now 1790572232662360 us; error 56 us; skew 500 ppm
I20260928 05:10:32.662747 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:32 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:32.665699 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:10:32.667693 636 webserver.cc:460] Webserver started at http://127.0.97.2:37285/ using document root <none> and password file <none>
I20260928 05:10:32.668467 636 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:32.668589 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:32.668903 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:32.671751 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "64a8d68fb9ae4916b6dc89c15b2e55cd"
format_stamp: "Formatted at 2026-09-28 05:10:32 on dist-test-slave-6718"
server_key: "b87c65d524dd4002a8f830d218a58a2d"
server_key_iv: "8a1c96fe9edee2086ea55992e042ff51"
server_key_version: "encryptionkey@0"
I20260928 05:10:32.672554 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "64a8d68fb9ae4916b6dc89c15b2e55cd"
format_stamp: "Formatted at 2026-09-28 05:10:32 on dist-test-slave-6718"
server_key: "b87c65d524dd4002a8f830d218a58a2d"
server_key_iv: "8a1c96fe9edee2086ea55992e042ff51"
server_key_version: "encryptionkey@0"
I20260928 05:10:32.677400 636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:10:32.680599 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:32.681869 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:10:32.682032 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "64a8d68fb9ae4916b6dc89c15b2e55cd"
format_stamp: "Formatted at 2026-09-28 05:10:32 on dist-test-slave-6718"
server_key: "b87c65d524dd4002a8f830d218a58a2d"
server_key_iv: "8a1c96fe9edee2086ea55992e042ff51"
server_key_version: "encryptionkey@0"
I20260928 05:10:32.682216 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:32.699708 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:32.703840 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:32.704114 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:32.705013 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:32.706281 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:32.706349 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:32.706476 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:32.706540 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:32.741003 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44215
I20260928 05:10:32.741046 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44215 every 8 connection(s)
I20260928 05:10:32.742317 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:10:32.748621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 636
I20260928 05:10:32.748808 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:10:32.749328 388 external_mini_cluster.cc:1428] Setting key 92564fff0ef76a2882d21af8328fa007
Sep 28 05:10:32 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:10:32.764981 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:32.744917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53433 (local address 127.0.97.62:43011)
0928 05:10:32.745106 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:32.745110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:32.746793 (+ 1683us) server_negotiation.cc:407] Connection header received
0928 05:10:32.749430 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:32.749437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:32.749527 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:32.749614 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:32.751536 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.752226 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.755148 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.755541 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.755635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:32.759124 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:32.759133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:32.759141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:32.759184 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:32.762014 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:32.762845 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:32.762851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:32.762854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:32.762984 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:32.763380 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:32.763385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:32.763388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:32.763633 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:32.763773 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:32.764436 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 05:10:32.764646 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:10:32.765947 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:32.745450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43011 (local address 127.0.97.2:53433)
0928 05:10:32.746165 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:32.746211 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:32.749081 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:32.749910 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:32.749924 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:32.750661 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:32.751365 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:32.751390 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.752373 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.752382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:32.755011 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:32.755027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:32.755765 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:32.755774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:32.756079 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:32.756764 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:32.756791 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:32.758896 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:32.762165 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:32.762174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:32.762191 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:32.762724 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:32.763088 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:32.763095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:32.763101 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:32.763268 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:32.763763 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:32.763771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:32.764089 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 05:10:32.764344 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 05:10:32.764862 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260928 05:10:32.767500 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43011
I20260928 05:10:32.767760 764 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:10:32.769109 764 heartbeater.cc:511] Master 127.0.97.62:43011 requested a full tablet report, sending...
I20260928 05:10:32.770794 440 ts_manager.cc:194] Registered new tserver with Master: 64a8d68fb9ae4916b6dc89c15b2e55cd (127.0.97.2:44215)
I20260928 05:10:32.771625 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:10:32.789664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43011
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:33.109872 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:33.110127 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:33.110220 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:33.117576 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:33.117676 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:33.117784 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:10:33.127184 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:33.128290 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:33.129695 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:33.144295 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:33.144373 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:33.144335 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:33.145259 769 server_base.cc:1034] running on GCE node
I20260928 05:10:33.145875 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:33.146922 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:33.148141 769 hybrid_clock.cc:648] HybridClock initialized: now 1790572233148078 us; error 56 us; skew 500 ppm
I20260928 05:10:33.148576 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:33.152191 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:10:33.153581 769 webserver.cc:460] Webserver started at http://127.0.97.3:44709/ using document root <none> and password file <none>
I20260928 05:10:33.154268 769 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:33.154389 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:33.154752 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:33.157552 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "11f7fbef522e4f36a3d3effe6f402b22"
format_stamp: "Formatted at 2026-09-28 05:10:33 on dist-test-slave-6718"
server_key: "acc0d1abb4f71f0af0493177aeddfd19"
server_key_iv: "9fd2845b1c271f2bc7e3e2364d13a352"
server_key_version: "encryptionkey@0"
I20260928 05:10:33.158339 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "11f7fbef522e4f36a3d3effe6f402b22"
format_stamp: "Formatted at 2026-09-28 05:10:33 on dist-test-slave-6718"
server_key: "acc0d1abb4f71f0af0493177aeddfd19"
server_key_iv: "9fd2845b1c271f2bc7e3e2364d13a352"
server_key_version: "encryptionkey@0"
I20260928 05:10:33.163341 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 05:10:33.167021 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:33.168555 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:10:33.168720 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "11f7fbef522e4f36a3d3effe6f402b22"
format_stamp: "Formatted at 2026-09-28 05:10:33 on dist-test-slave-6718"
server_key: "acc0d1abb4f71f0af0493177aeddfd19"
server_key_iv: "9fd2845b1c271f2bc7e3e2364d13a352"
server_key_version: "encryptionkey@0"
I20260928 05:10:33.168922 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:33.189380 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:33.193686 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:33.193961 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:33.195005 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:33.196290 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:33.196360 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:33.196483 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:33.196532 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:33.231057 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44713
I20260928 05:10:33.231135 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44713 every 8 connection(s)
I20260928 05:10:33.232398 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:10:33.235383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 769
I20260928 05:10:33.235579 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:10:33.236110 388 external_mini_cluster.cc:1428] Setting key 86eafb819edd3520da631b5d84f7d733
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:33.253489 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.235416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35663 (local address 127.0.97.62:43011)
0928 05:10:33.235623 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:33.235629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:33.237090 (+ 1461us) server_negotiation.cc:407] Connection header received
0928 05:10:33.239523 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:33.239526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:33.239598 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:33.239688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:33.241279 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.242020 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.243707 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.244156 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.244258 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.247772 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:33.247776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:33.247779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:33.247812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:33.250168 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.250976 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.250986 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.250988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.251156 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.251794 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.251799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.251802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.252146 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:33.252290 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:33.253099 (+ 809us) server_negotiation.cc:299] Negotiation successful
0928 05:10:33.253256 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:10:33.254663 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.235992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43011 (local address 127.0.97.3:35663)
0928 05:10:33.236897 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:33.236956 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:33.239214 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:33.239982 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:33.239995 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:33.240517 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:33.241115 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:33.241133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.242170 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.242174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:33.243537 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:33.243548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.244350 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.244355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:33.244545 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.245295 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:33.245316 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:33.247533 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:33.250324 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:33.250333 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:33.250352 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:33.250844 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:33.251294 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:33.251299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:33.251302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:33.251566 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:33.252270 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:33.252276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:33.252746 (+ 470us) client_negotiation.cc:769] Sending connection context
0928 05:10:33.252973 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 05:10:33.253502 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":347,"threads_started":1}
I20260928 05:10:33.256609 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43011
I20260928 05:10:33.256942 897 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:33.257670 897 heartbeater.cc:511] Master 127.0.97.62:43011 requested a full tablet report, sending...
I20260928 05:10:33.259110 440 ts_manager.cc:194] Registered new tserver with Master: 11f7fbef522e4f36a3d3effe6f402b22 (127.0.97.3:44713)
I20260928 05:10:33.259860 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35663
I20260928 05:10:33.264633 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:10:33.273218 630 heartbeater.cc:503] Master 127.0.97.62:43011 was elected leader, sending a full tablet report...
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:33.310453 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.294122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.0.97.1:41647)
0928 05:10:33.294827 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:33.294834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:33.294881 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:10:33.295071 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:33.295080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:33.295331 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:33.295485 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:33.296910 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.298758 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.300012 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.300558 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.300701 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.304972 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:33.304983 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:33.304999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:33.305032 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:33.308004 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.308633 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.308640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.308642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.308770 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.309090 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.309093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.309095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.309423 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:33.309658 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:33.309846 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 05:10:33.310033 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":331,"threads_started":1}
W20260928 05:10:33.311869 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48888
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:33.354669 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.340729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.97.1:41647)
0928 05:10:33.340953 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:33.340960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:33.341213 (+ 253us) server_negotiation.cc:407] Connection header received
0928 05:10:33.341434 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:33.341438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:33.341526 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:33.341630 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:33.343099 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.343813 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.344989 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.345378 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.345505 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.349231 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:33.349236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:33.349239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:33.349272 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:33.352177 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.352806 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.352817 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.352821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.352932 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.353266 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.353273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.353276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.353521 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:33.353758 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:33.354075 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 05:10:33.354296 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:10:33.356247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 502
I20260928 05:10:33.368037 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.357221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.0.97.1:41647)
0928 05:10:33.357368 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:33.357372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:33.357710 (+ 338us) server_negotiation.cc:407] Connection header received
0928 05:10:33.357826 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:33.357830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:33.357883 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:33.357956 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:33.359239 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.359907 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.361079 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.361453 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.361538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.363680 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:33.363684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:33.363687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:33.363711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:33.365908 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.366477 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.366480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.366483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.366662 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.366957 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.366960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.366962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.367178 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:33.367347 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:33.367653 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 05:10:33.367795 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:10:33.368721 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:33.450791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 502
I20260928 05:10:33.466768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 636
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:10:33.482483 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.467615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.0.97.2:44215)
0928 05:10:33.468224 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:33.468232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:33.468258 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:10:33.468321 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:33.468328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:33.468549 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:33.468684 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:33.469767 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.470883 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.472066 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.472625 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.472806 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.476644 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:33.476655 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:33.476676 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:33.476712 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:33.479506 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.480317 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.480324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.480327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.480470 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.480910 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.480917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.480921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.481269 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:33.481569 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:33.481881 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 05:10:33.482153 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":358,"threads_started":1}
I20260928 05:10:33.483767 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:33.566325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 636
I20260928 05:10:33.590731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 769
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:10:33.605527 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:33.591602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.97.3:44713)
0928 05:10:33.592153 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:33.592160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:33.592188 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:10:33.592257 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:33.592263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:33.592501 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:33.592619 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:33.593785 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.594857 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.596037 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:33.596568 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:33.596659 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:33.600558 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:33.600569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:33.600584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:33.600619 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:33.603213 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.603781 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.603787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.603789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.603895 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:33.604186 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:33.604189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:33.604191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:33.604495 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:33.604704 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:33.604967 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 05:10:33.605149 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":268,"threads_started":1}
I20260928 05:10:33.606819 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:33.689134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 769
I20260928 05:10:33.706163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 408
I20260928 05:10:33.707366 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:33.807915 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 408
2026-09-28T05:10:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5092 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[918](info): set up 2 sockets
Sep 28 05:10:33 dist-test-slave-6718 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260928 05:10:34.890420 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/test-user.keytab.
W20260928 05:10:35.375638 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.407s user 0.000s sys 0.004s
Sep 28 05:10:35 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260928 05:10:36.162091 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.739s user 0.000s sys 0.007s
I20260928 05:10:36.163379 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45777
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:36.473941 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:36.474247 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:36.474339 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:36.481716 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:36.481832 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:36.481879 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:36.481937 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:36.481981 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:36.491946 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:36.493208 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:36.494496 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:36.506502 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:36.506577 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:36.506600 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:36.507411 931 server_base.cc:1034] running on GCE node
I20260928 05:10:36.508221 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:36.509528 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:36.510784 931 hybrid_clock.cc:648] HybridClock initialized: now 1790572236510731 us; error 54 us; skew 500 ppm
I20260928 05:10:36.511130 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:36 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:36.514022 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:36.515499 931 webserver.cc:460] Webserver started at http://127.0.97.62:36657/ using document root <none> and password file <none>
I20260928 05:10:36.516161 931 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:36.516335 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:36.516630 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:36.519513 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "e7a94a250ed44ca5a69b3526a6f39d85"
format_stamp: "Formatted at 2026-09-28 05:10:36 on dist-test-slave-6718"
server_key: "e433fd546f8225dfac248baecbf554cb"
server_key_iv: "b46bd33bd2ce6ec011877a02a6fc2d76"
server_key_version: "encryptionkey@0"
I20260928 05:10:36.520252 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7a94a250ed44ca5a69b3526a6f39d85"
format_stamp: "Formatted at 2026-09-28 05:10:36 on dist-test-slave-6718"
server_key: "e433fd546f8225dfac248baecbf554cb"
server_key_iv: "b46bd33bd2ce6ec011877a02a6fc2d76"
server_key_version: "encryptionkey@0"
I20260928 05:10:36.525646 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:10:36.528993 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:36.530244 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:10:36.530417 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "e7a94a250ed44ca5a69b3526a6f39d85"
format_stamp: "Formatted at 2026-09-28 05:10:36 on dist-test-slave-6718"
server_key: "e433fd546f8225dfac248baecbf554cb"
server_key_iv: "b46bd33bd2ce6ec011877a02a6fc2d76"
server_key_version: "encryptionkey@0"
I20260928 05:10:36.530661 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:36.558063 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:36.562359 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:36.562714 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:36.580850 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45777
I20260928 05:10:36.580880 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45777 every 8 connection(s)
I20260928 05:10:36.582161 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:10:36.585799 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:36.589311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 931
I20260928 05:10:36.589529 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:10:36.590330 388 external_mini_cluster.cc:1428] Setting key ce19d77e45a80ff5860ea184e1df7ee1
I20260928 05:10:36.600334 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85: Bootstrap starting.
I20260928 05:10:36.604015 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:36.605013 998 log.cc:824] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:10:36 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:36.608280 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85: No bootstrap required, opened a new log
I20260928 05:10:36.615015 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:36.592798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.0.97.62:45777)
0928 05:10:36.593722 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:36.593740 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:36.594228 (+ 488us) server_negotiation.cc:407] Connection header received
0928 05:10:36.596109 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:36.596141 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:36.596704 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:36.597150 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:36.598496 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:36.600477 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:36.601690 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:36.602786 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:36.602961 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:36.606912 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:36.606921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:36.606939 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:36.606983 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:36.610773 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:36.611384 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:36.611394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:36.611405 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:36.611600 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:36.611919 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:36.611924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:36.611927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:36.612593 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:36.612870 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:36.613219 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 05:10:36.613641 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":408,"threads_started":1}
I20260928 05:10:36.618582 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45777 } }
I20260928 05:10:36.618908 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:36.618980 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a94a250ed44ca5a69b3526a6f39d85, State: Initialized, Role: FOLLOWER
I20260928 05:10:36.619451 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45777 } }
I20260928 05:10:36.619643 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:10:36.619740 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:10:36.619858 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:36.633378 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45777 } }
I20260928 05:10:36.634152 998 leader_election.cc:302] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a94a250ed44ca5a69b3526a6f39d85; no voters:
I20260928 05:10:36.634920 998 leader_election.cc:288] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:10:36.635033 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:36.635701 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [term 1 LEADER]: Becoming Leader. State: Replica: e7a94a250ed44ca5a69b3526a6f39d85, State: Running, Role: LEADER
I20260928 05:10:36.636332 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45777 } }
I20260928 05:10:36.638823 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:10:36.639247 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45777 } } }
I20260928 05:10:36.639604 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:36.641769 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a94a250ed44ca5a69b3526a6f39d85. Latest consensus state: current_term: 1 leader_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a94a250ed44ca5a69b3526a6f39d85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45777 } } }
I20260928 05:10:36.642120 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:36.643615 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:10:36.649166 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:10:36.658056 1007 catalog_manager.cc:1347] Generated new cluster ID: 31a89a36bef34f3daf4be056a4ee6e7a
I20260928 05:10:36.658190 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:10:36.675551 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:10:36.676898 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:10:36.687737 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a94a250ed44ca5a69b3526a6f39d85: Generated new TSK 0
I20260928 05:10:36.688593 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260928 05:10:36.820739 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.127s user 0.001s sys 0.008s
I20260928 05:10:36.821965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45777
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:37.137036 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:37.137295 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:37.137478 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:37.145130 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:37.145260 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:37.145440 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:10:37.155839 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:37.157179 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:37.158720 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:37.171432 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:37.171497 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:37.171522 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:37.172303 1024 server_base.cc:1034] running on GCE node
I20260928 05:10:37.172910 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:37.173836 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:37.175052 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790572237174994 us; error 53 us; skew 500 ppm
I20260928 05:10:37.175356 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:37 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:37.178402 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:10:37.179919 1024 webserver.cc:460] Webserver started at http://127.0.97.1:41645/ using document root <none> and password file <none>
I20260928 05:10:37.180604 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:37.180753 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:37.181063 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:37.183961 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "96d6683e3dae4897b6decb4210bbd43a"
format_stamp: "Formatted at 2026-09-28 05:10:37 on dist-test-slave-6718"
server_key: "b9ca13f4587bb6bd72433802ac8b0fa3"
server_key_iv: "e1bc4e4116d62a046b1b5191b09cdcf0"
server_key_version: "encryptionkey@0"
I20260928 05:10:37.184741 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "96d6683e3dae4897b6decb4210bbd43a"
format_stamp: "Formatted at 2026-09-28 05:10:37 on dist-test-slave-6718"
server_key: "b9ca13f4587bb6bd72433802ac8b0fa3"
server_key_iv: "e1bc4e4116d62a046b1b5191b09cdcf0"
server_key_version: "encryptionkey@0"
I20260928 05:10:37.190061 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:10:37.193496 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:37.194948 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:10:37.195127 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "96d6683e3dae4897b6decb4210bbd43a"
format_stamp: "Formatted at 2026-09-28 05:10:37 on dist-test-slave-6718"
server_key: "b9ca13f4587bb6bd72433802ac8b0fa3"
server_key_iv: "e1bc4e4116d62a046b1b5191b09cdcf0"
server_key_version: "encryptionkey@0"
I20260928 05:10:37.195333 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:37.218400 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:37.222513 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:37.222862 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:37.223836 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:37.225176 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:37.225288 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:37.225447 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:37.225520 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:37.259094 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41517
I20260928 05:10:37.259151 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41517 every 8 connection(s)
I20260928 05:10:37.260458 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:10:37.270098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1024
I20260928 05:10:37.270298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:10:37.271083 388 external_mini_cluster.cc:1428] Setting key 93e039de72519c975869122886a12589
Sep 28 05:10:37 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:37.280233 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:37.263004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33149 (local address 127.0.97.62:45777)
0928 05:10:37.263452 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:37.263458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:37.264450 (+ 992us) server_negotiation.cc:407] Connection header received
0928 05:10:37.266138 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:37.266143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:37.266208 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:37.266304 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:37.267751 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.268504 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.269937 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.270597 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.270745 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:37.274220 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:37.274224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:37.274228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:37.274255 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:37.276927 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:37.277755 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:37.277764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:37.277769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:37.277894 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:37.278354 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:37.278362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:37.278365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:37.278732 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:37.278886 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:37.279601 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 05:10:37.279840 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260928 05:10:37.281230 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:37.263595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45777 (local address 127.0.97.1:33149)
0928 05:10:37.264283 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:37.264330 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:37.265889 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:37.266495 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:37.266503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:37.267050 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:37.267581 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:37.267597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.268626 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.268630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:37.269790 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:37.269801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.270803 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.270808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:37.270987 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:37.271697 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:37.271718 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:37.274001 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:37.277094 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:37.277104 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:37.277121 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:37.277567 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:37.278024 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:37.278029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:37.278033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:37.278227 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:37.278876 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:37.278884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:37.279220 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 05:10:37.279502 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 05:10:37.280063 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":252,"threads_started":1}
I20260928 05:10:37.282943 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45777
I20260928 05:10:37.283274 1152 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:37.284175 1152 heartbeater.cc:511] Master 127.0.97.62:45777 requested a full tablet report, sending...
I20260928 05:10:37.286706 963 ts_manager.cc:194] Registered new tserver with Master: 96d6683e3dae4897b6decb4210bbd43a (127.0.97.1:41517)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:10:37.288304 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:10:37.310006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45777
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:37.620373 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:37.620633 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:37.620823 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:37.628010 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:37.628129 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:37.628328 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:10:37.638343 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:37.639899 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:37.641196 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:37.653898 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:37.653894 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:37.653925 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:37.655511 1158 server_base.cc:1034] running on GCE node
I20260928 05:10:37.656111 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:37.657003 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:37.658224 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790572237658183 us; error 54 us; skew 500 ppm
I20260928 05:10:37.658566 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:37 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:37.661564 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:10:37.663152 1158 webserver.cc:460] Webserver started at http://127.0.97.2:46823/ using document root <none> and password file <none>
I20260928 05:10:37.663849 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:37.663992 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:37.664306 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:37.667162 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46"
format_stamp: "Formatted at 2026-09-28 05:10:37 on dist-test-slave-6718"
server_key: "179a3ff3059fe3e27fb93a14236e22a0"
server_key_iv: "4c79f6e99a986b212e01e772ffbbc84e"
server_key_version: "encryptionkey@0"
I20260928 05:10:37.667917 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46"
format_stamp: "Formatted at 2026-09-28 05:10:37 on dist-test-slave-6718"
server_key: "179a3ff3059fe3e27fb93a14236e22a0"
server_key_iv: "4c79f6e99a986b212e01e772ffbbc84e"
server_key_version: "encryptionkey@0"
I20260928 05:10:37.672863 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 05:10:37.676060 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:37.677274 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:10:37.677439 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46"
format_stamp: "Formatted at 2026-09-28 05:10:37 on dist-test-slave-6718"
server_key: "179a3ff3059fe3e27fb93a14236e22a0"
server_key_iv: "4c79f6e99a986b212e01e772ffbbc84e"
server_key_version: "encryptionkey@0"
I20260928 05:10:37.677639 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:37.695107 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:37.699075 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:37.699357 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:37.700311 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:37.701568 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:37.701670 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:37.701824 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:37.701897 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:37.733577 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43779
I20260928 05:10:37.733620 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43779 every 8 connection(s)
I20260928 05:10:37.734994 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:10:37.735921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1158
I20260928 05:10:37.736085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:10:37.736636 388 external_mini_cluster.cc:1428] Setting key 3db015d92fb5c9c85593103e0944088a
Sep 28 05:10:37 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:10:37.758298 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:37.737773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59415 (local address 127.0.97.62:45777)
0928 05:10:37.737977 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:37.737986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:37.739830 (+ 1844us) server_negotiation.cc:407] Connection header received
0928 05:10:37.741647 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:37.741656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:37.741761 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:37.741874 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:37.743635 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.744520 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.747234 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.747720 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.747854 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:37.752498 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:37.752503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:37.752506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:37.752541 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:37.755252 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:37.756015 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:37.756019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:37.756021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:37.756134 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:37.756610 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:37.756615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:37.756618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:37.756881 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:37.757031 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:37.757726 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 05:10:37.757944 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:10:37.759315 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:37.738568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45777 (local address 127.0.97.2:59415)
0928 05:10:37.739668 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:37.739714 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:37.741353 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:37.742074 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:37.742082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:37.742681 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:37.743453 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:37.743480 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.744688 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.744697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:37.747059 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:37.747077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:37.747984 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:37.747995 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:37.748342 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:37.749123 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:37.749156 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:37.752202 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:37.755379 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:37.755386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:37.755401 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:37.755902 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:37.756225 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:37.756228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:37.756231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:37.756488 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:37.757006 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:37.757014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:37.757363 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 05:10:37.757638 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 05:10:37.758176 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":401,"threads_started":1}
I20260928 05:10:37.760881 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45777
I20260928 05:10:37.761178 1286 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:37.761904 1286 heartbeater.cc:511] Master 127.0.97.62:45777 requested a full tablet report, sending...
I20260928 05:10:37.763384 963 ts_manager.cc:194] Registered new tserver with Master: f7e2e2f8bcd947e3bedb6a01394f6b46 (127.0.97.2:43779)
I20260928 05:10:37.764194 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:10:37.777595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45777
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:38.095844 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:38.096113 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:38.096277 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:38.103398 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:38.103506 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:38.103727 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:10:38.115875 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:38.117187 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:38.118513 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:38.133366 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:38.133411 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:38.133322 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:38.136015 1291 server_base.cc:1034] running on GCE node
I20260928 05:10:38.136646 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:38.137914 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:38.139151 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790572238139111 us; error 53 us; skew 500 ppm
I20260928 05:10:38.139469 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:38.142655 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:10:38.144134 1291 webserver.cc:460] Webserver started at http://127.0.97.3:43283/ using document root <none> and password file <none>
I20260928 05:10:38.144835 1291 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:38.145004 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:38.145649 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:38.148581 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "fbb87e3ea7ac4be8874a1df348b4912e"
format_stamp: "Formatted at 2026-09-28 05:10:38 on dist-test-slave-6718"
server_key: "73bef9be32c2818f154a0cd3631c4210"
server_key_iv: "43a75175307347de1a860fd4803c1609"
server_key_version: "encryptionkey@0"
I20260928 05:10:38.149336 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fbb87e3ea7ac4be8874a1df348b4912e"
format_stamp: "Formatted at 2026-09-28 05:10:38 on dist-test-slave-6718"
server_key: "73bef9be32c2818f154a0cd3631c4210"
server_key_iv: "43a75175307347de1a860fd4803c1609"
server_key_version: "encryptionkey@0"
I20260928 05:10:38.155148 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:10:38.158902 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:38.160817 1291 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:10:38.160987 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "fbb87e3ea7ac4be8874a1df348b4912e"
format_stamp: "Formatted at 2026-09-28 05:10:38 on dist-test-slave-6718"
server_key: "73bef9be32c2818f154a0cd3631c4210"
server_key_iv: "43a75175307347de1a860fd4803c1609"
server_key_version: "encryptionkey@0"
I20260928 05:10:38.161258 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:38.175165 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:38.179085 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:38.179324 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:38.180193 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:38.181532 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:38.181609 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:38.181696 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:38.181742 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:38.214465 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44585
I20260928 05:10:38.214509 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44585 every 8 connection(s)
I20260928 05:10:38.215763 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:10:38.224130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1291
I20260928 05:10:38.224316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:10:38.224756 388 external_mini_cluster.cc:1428] Setting key 5994d39418e8aba53f6026f94936683a
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:38.235349 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.218181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53485 (local address 127.0.97.62:45777)
0928 05:10:38.218365 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.218369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.219490 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 05:10:38.221231 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.221237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.221325 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.221431 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.223246 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.223987 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.225807 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.226335 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.226469 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.229861 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.229866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.229869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.229898 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.232366 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.233078 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.233088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.233090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.233210 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.233626 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.233631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.233634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.233893 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.234076 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.234805 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.235010 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:10:38.236243 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.218663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45777 (local address 127.0.97.3:53485)
0928 05:10:38.219331 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.219367 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:38.220973 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:38.221658 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:38.221666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:38.222552 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:38.223067 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.223083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.224128 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.224134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.225665 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.225677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.226612 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.226620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.226877 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.227560 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:38.227591 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:38.229655 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:38.232505 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:38.232514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:38.232530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:38.232953 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:38.233317 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:38.233322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:38.233326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:38.233505 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:38.234021 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:38.234029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:38.234367 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 05:10:38.234647 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 05:10:38.235155 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":258,"threads_started":1}
I20260928 05:10:38.237522 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45777
I20260928 05:10:38.237811 1419 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:38.238683 1419 heartbeater.cc:511] Master 127.0.97.62:45777 requested a full tablet report, sending...
I20260928 05:10:38.240170 963 ts_manager.cc:194] Registered new tserver with Master: fbb87e3ea7ac4be8874a1df348b4912e (127.0.97.3:44585)
I20260928 05:10:38.240996 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53485
I20260928 05:10:38.252153 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:10:38.270807 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.258635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.0.97.62:45777)
0928 05:10:38.258827 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.258836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.259202 (+ 366us) server_negotiation.cc:407] Connection header received
0928 05:10:38.259406 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.259413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.259478 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.259580 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.260999 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.261672 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.263077 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.263491 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.263624 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.265983 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.265988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.265991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.266019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.268320 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.268931 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.268938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.268946 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.269060 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.269416 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.269422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.269425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.269699 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.269950 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.270268 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.270485 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 05:10:38.276856 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:10:38.284029 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:38.291301 1152 heartbeater.cc:503] Master 127.0.97.62:45777 was elected leader, sending a full tablet report...
I20260928 05:10:38.308936 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.296277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44585 (local address 127.0.0.1:33092)
0928 05:10:38.300504 (+ 4227us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.300528 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:38.300706 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:38.301669 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:38.301676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:38.301703 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:38.302085 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.302098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.303874 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.303881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.306519 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.306560 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.307888 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.307895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.308088 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.308150 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 05:10:38.308289 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 05:10:38.308469 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4058,"mutex_wait_us":79,"thread_start_us":3897,"threads_started":1}
I20260928 05:10:38.309932 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.296932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.0.97.3:44585)
0928 05:10:38.300002 (+ 3070us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.300014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.300871 (+ 857us) server_negotiation.cc:407] Connection header received
0928 05:10:38.301005 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.301017 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.301351 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.301584 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:38.302300 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.303747 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.306723 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.307689 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.307840 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.308077 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.309417 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.309581 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2854,"thread_start_us":2293,"threads_started":1}
I20260928 05:10:38.309465 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.299060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43779 (local address 127.0.0.1:34338)
0928 05:10:38.300816 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.300833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:38.300945 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:38.302047 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:38.302055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:38.302078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:38.302484 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.302496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.304939 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.304948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.306757 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.306772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.308832 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.308840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.309010 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.309042 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:10:38.309105 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 05:10:38.309206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"mutex_wait_us":77,"thread_start_us":1459,"threads_started":1}
I20260928 05:10:38.310307 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.299653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.0.97.2:43779)
0928 05:10:38.301460 (+ 1807us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.301468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.301493 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:10:38.301586 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.301596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.301826 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.302691 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:38.302911 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.304769 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.307249 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.308604 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.309566 (+ 962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.309742 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.309818 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.309974 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1655,"thread_start_us":1748,"threads_started":1}
I20260928 05:10:38.310557 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.296043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41517 (local address 127.0.0.1:40726)
0928 05:10:38.298025 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.298087 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:38.298296 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:38.302844 (+ 4548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:38.302849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:38.302873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:38.303162 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.303172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.304976 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.304982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.307680 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.307693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.309786 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.309792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.309950 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.309978 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 05:10:38.310071 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:10:38.310203 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1612,"thread_start_us":1411,"threads_started":1}
I20260928 05:10:38.311141 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.296334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.0.97.1:41517)
0928 05:10:38.302339 (+ 6005us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.302345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.302364 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:10:38.302431 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.302436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.302650 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.302850 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:38.303354 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.304842 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.308172 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.309584 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.309719 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.309929 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.310684 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.310841 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5863,"thread_start_us":5702,"threads_started":1}
I20260928 05:10:38.313877 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4249e0e7f5ae46abadb5ad06888b235c (DEFAULT_TABLE table=test-table [id=50bb768d6ad6410b8f5aeb2633281c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:38.313966 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 4249e0e7f5ae46abadb5ad06888b235c (DEFAULT_TABLE table=test-table [id=50bb768d6ad6410b8f5aeb2633281c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:38.314283 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4249e0e7f5ae46abadb5ad06888b235c (DEFAULT_TABLE table=test-table [id=50bb768d6ad6410b8f5aeb2633281c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:38.315214 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4249e0e7f5ae46abadb5ad06888b235c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:38.315367 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4249e0e7f5ae46abadb5ad06888b235c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:38.315476 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4249e0e7f5ae46abadb5ad06888b235c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:38.328418 1437 tablet_bootstrap.cc:492] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: Bootstrap starting.
I20260928 05:10:38.330152 1438 tablet_bootstrap.cc:492] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: Bootstrap starting.
I20260928 05:10:38.331224 1439 tablet_bootstrap.cc:492] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: Bootstrap starting.
I20260928 05:10:38.333210 1437 tablet_bootstrap.cc:654] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:38.333416 1438 tablet_bootstrap.cc:654] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:38.334127 1439 tablet_bootstrap.cc:654] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:38.334944 1437 log.cc:824] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:38.334944 1438 log.cc:824] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:38.335193 1439 log.cc:824] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:38.341413 1439 tablet_bootstrap.cc:492] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: No bootstrap required, opened a new log
I20260928 05:10:38.341482 1438 tablet_bootstrap.cc:492] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: No bootstrap required, opened a new log
I20260928 05:10:38.341480 1437 tablet_bootstrap.cc:492] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: No bootstrap required, opened a new log
I20260928 05:10:38.341683 1439 ts_tablet_manager.cc:1397] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 05:10:38.341743 1437 ts_tablet_manager.cc:1397] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260928 05:10:38.341729 1438 ts_tablet_manager.cc:1397] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260928 05:10:38.349603 1439 raft_consensus.cc:348] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.350039 1439 raft_consensus.cc:374] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:38.350131 1439 raft_consensus.cc:729] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb87e3ea7ac4be8874a1df348b4912e, State: Initialized, Role: FOLLOWER
I20260928 05:10:38.350786 1439 consensus_queue.cc:260] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.351840 1419 heartbeater.cc:503] Master 127.0.97.62:45777 was elected leader, sending a full tablet report...
I20260928 05:10:38.352528 1439 ts_tablet_manager.cc:1428] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 05:10:38.352844 1438 raft_consensus.cc:348] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.352844 1437 raft_consensus.cc:348] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.353259 1437 raft_consensus.cc:374] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:38.353289 1438 raft_consensus.cc:374] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:38.353369 1437 raft_consensus.cc:729] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e2e2f8bcd947e3bedb6a01394f6b46, State: Initialized, Role: FOLLOWER
I20260928 05:10:38.353375 1438 raft_consensus.cc:729] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d6683e3dae4897b6decb4210bbd43a, State: Initialized, Role: FOLLOWER
I20260928 05:10:38.353952 1438 consensus_queue.cc:260] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.353952 1437 consensus_queue.cc:260] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.355758 1437 ts_tablet_manager.cc:1428] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260928 05:10:38.356215 1438 ts_tablet_manager.cc:1428] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 05:10:38.356724 1286 heartbeater.cc:503] Master 127.0.97.62:45777 was elected leader, sending a full tablet report...
I20260928 05:10:38.367062 1443 raft_consensus.cc:482] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:38.367321 1443 raft_consensus.cc:504] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.368832 1443 leader_election.cc:288] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96d6683e3dae4897b6decb4210bbd43a (127.0.97.1:41517), f7e2e2f8bcd947e3bedb6a01394f6b46 (127.0.97.2:43779)
I20260928 05:10:38.374354 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.369061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41517 (local address 127.0.97.3:45753)
0928 05:10:38.369651 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.369676 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:38.369810 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:38.370120 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:38.370124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:38.370155 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:38.370338 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.370346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.371346 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.371353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.372631 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.372642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.373674 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.373680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.373845 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.373866 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:10:38.373963 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:10:38.374076 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":42}
I20260928 05:10:38.374353 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.369626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45753 (local address 127.0.97.1:41517)
0928 05:10:38.369792 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.369796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.369813 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:10:38.369939 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.369942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.369994 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.370101 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:38.370468 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.371213 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.372784 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.373520 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.373654 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.373743 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.373985 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.374101 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:10:38.375371 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.369274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38091 (local address 127.0.97.2:43779)
0928 05:10:38.369450 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.369457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.370565 (+ 1108us) server_negotiation.cc:407] Connection header received
0928 05:10:38.370737 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.370742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.370804 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.370893 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:38.371348 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.371996 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.373587 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.374332 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.374485 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.374620 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.375019 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.375148 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:10:38.375465 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.369609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43779 (local address 127.0.97.3:38091)
0928 05:10:38.370448 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.370461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:38.370614 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:38.370904 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:38.370908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:38.370928 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:38.371214 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.371222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.372133 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.372138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.373437 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:38.373447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.374686 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.374692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:38.374860 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.374895 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:10:38.374992 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:10:38.375115 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":1,"thread_start_us":482,"threads_started":1}
I20260928 05:10:38.375430 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4249e0e7f5ae46abadb5ad06888b235c" candidate_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96d6683e3dae4897b6decb4210bbd43a" is_pre_election: true
I20260928 05:10:38.376029 1107 raft_consensus.cc:2454] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb87e3ea7ac4be8874a1df348b4912e in term 0.
I20260928 05:10:38.376513 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4249e0e7f5ae46abadb5ad06888b235c" candidate_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" is_pre_election: true
I20260928 05:10:38.376705 1307 leader_election.cc:302] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96d6683e3dae4897b6decb4210bbd43a, fbb87e3ea7ac4be8874a1df348b4912e; no voters:
I20260928 05:10:38.377430 1443 raft_consensus.cc:2781] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:38.376978 1241 raft_consensus.cc:2454] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb87e3ea7ac4be8874a1df348b4912e in term 0.
I20260928 05:10:38.377568 1443 raft_consensus.cc:482] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:38.377707 1443 raft_consensus.cc:3037] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:38.379899 1443 raft_consensus.cc:504] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.380395 1443 leader_election.cc:288] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [CANDIDATE]: Term 1 election: Requested vote from peers 96d6683e3dae4897b6decb4210bbd43a (127.0.97.1:41517), f7e2e2f8bcd947e3bedb6a01394f6b46 (127.0.97.2:43779)
I20260928 05:10:38.380761 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4249e0e7f5ae46abadb5ad06888b235c" candidate_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96d6683e3dae4897b6decb4210bbd43a"
I20260928 05:10:38.380798 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4249e0e7f5ae46abadb5ad06888b235c" candidate_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46"
I20260928 05:10:38.380976 1107 raft_consensus.cc:3037] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:38.380983 1241 raft_consensus.cc:3037] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:38.383262 1107 raft_consensus.cc:2454] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb87e3ea7ac4be8874a1df348b4912e in term 1.
I20260928 05:10:38.383318 1241 raft_consensus.cc:2454] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb87e3ea7ac4be8874a1df348b4912e in term 1.
I20260928 05:10:38.383682 1307 leader_election.cc:302] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96d6683e3dae4897b6decb4210bbd43a, fbb87e3ea7ac4be8874a1df348b4912e; no voters:
I20260928 05:10:38.383942 1443 raft_consensus.cc:2781] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:38.384320 1443 raft_consensus.cc:686] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [term 1 LEADER]: Becoming Leader. State: Replica: fbb87e3ea7ac4be8874a1df348b4912e, State: Running, Role: LEADER
I20260928 05:10:38.384727 1443 consensus_queue.cc:237] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } }
I20260928 05:10:38.388999 963 catalog_manager.cc:5495] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb87e3ea7ac4be8874a1df348b4912e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96d6683e3dae4897b6decb4210bbd43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb87e3ea7ac4be8874a1df348b4912e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7e2e2f8bcd947e3bedb6a01394f6b46" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43779 } health_report { overall_health: UNKNOWN } } }
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:38.411204 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.397588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.0.97.1:41517)
0928 05:10:38.397771 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.397776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.398296 (+ 520us) server_negotiation.cc:407] Connection header received
0928 05:10:38.398420 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.398424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.398482 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.398638 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.400056 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.400758 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.401882 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.402271 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.402358 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.406202 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.406213 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.406227 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.406262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.409054 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.409610 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.409616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.409618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.409727 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.410016 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.410020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.410023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.410343 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.410610 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.410814 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.410958 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:38.455394 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.442149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.0.97.1:41517)
0928 05:10:38.442413 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.442420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.442651 (+ 231us) server_negotiation.cc:407] Connection header received
0928 05:10:38.442813 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.442817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.442879 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.442972 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.444224 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.445045 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.446240 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.446686 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.446779 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.450380 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.450385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.450388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.450418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.452991 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.453606 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.453610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.453612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.453736 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.454073 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.454076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.454078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.454385 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.454685 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.454880 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.455037 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260928 05:10:38.467305 1420 tablet.cc:2369] T 4249e0e7f5ae46abadb5ad06888b235c P fbb87e3ea7ac4be8874a1df348b4912e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260928 05:10:38.486913 1287 tablet.cc:2369] T 4249e0e7f5ae46abadb5ad06888b235c P f7e2e2f8bcd947e3bedb6a01394f6b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:38.499111 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.485328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.97.1:41517)
0928 05:10:38.485580 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.485588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.485870 (+ 282us) server_negotiation.cc:407] Connection header received
0928 05:10:38.486113 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.486117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.486181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.486304 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.487745 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.488428 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.489641 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.490065 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.490195 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.494178 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.494183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.494186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.494234 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.496622 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.497252 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.497256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.497258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.497370 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.497724 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.497731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.497737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.497998 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.498253 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.498521 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.498766 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 05:10:38.500734 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4249e0e7f5ae46abadb5ad06888b235c: Uninitialized: clean time has not yet been initialized
I20260928 05:10:38.503520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1024
W20260928 05:10:38.512835 1153 tablet.cc:2369] T 4249e0e7f5ae46abadb5ad06888b235c P 96d6683e3dae4897b6decb4210bbd43a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:10:38.517323 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.504356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.97.1:41517)
0928 05:10:38.504762 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.504766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.507008 (+ 2242us) server_negotiation.cc:407] Connection header received
0928 05:10:38.507129 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.507133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.507196 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.507272 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.508295 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.508942 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.510062 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.510429 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.510594 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.512644 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.512650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.512654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.512691 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.515341 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.515904 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.515908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.515910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.516014 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.516314 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.516317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.516319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.516560 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.516766 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.516967 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.517105 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260928 05:10:38.517917 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:38.605847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1024
I20260928 05:10:38.622234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1158
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:10:38.636659 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.623150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.97.2:43779)
0928 05:10:38.623345 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.623353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.623375 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:10:38.623444 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.623447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.623524 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.623646 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.624781 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.625505 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.626822 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.627198 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.627323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.631112 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.631126 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.631141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.631172 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.633925 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.634679 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.634691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.634694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.634817 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.635220 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.635229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.635232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.635588 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.635917 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.636119 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.636340 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:10:38.637355 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:38.724352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1158
I20260928 05:10:38.741544 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1291
Sep 28 05:10:38 dist-test-slave-6718 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:10:38.755398 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.742208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.0.97.3:44585)
0928 05:10:38.742374 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:38.742378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:38.742440 (+ 62us) server_negotiation.cc:407] Connection header received
0928 05:10:38.742619 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:38.742623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:38.742683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:38.742779 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:38.743999 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.744748 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.745914 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:38.746290 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:38.746379 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:38.750305 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:38.750318 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:38.750334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:38.750367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:38.753084 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.753637 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.753644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.753647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.753782 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:38.754075 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:38.754078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:38.754080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:38.754389 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:38.754747 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:38.754874 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 05:10:38.755039 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:10:38.756050 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:38.845513 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1291
I20260928 05:10:38.845559 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:38.845051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41517 (local address 127.0.97.3:47367)
0928 05:10:38.845242 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:38.845392 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 05:10:38.861737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 931
I20260928 05:10:38.862885 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:38.968132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 931
2026-09-28T05:10:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5160 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:39 dist-test-slave-6718 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:39 dist-test-slave-6718 krb5kdc[1466](info): set up 2 sockets
Sep 28 05:10:39 dist-test-slave-6718 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260928 05:10:40.034452 388 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.011s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:10:40 dist-test-slave-6718 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:10:40.211793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39459
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:40.518364 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:40.518698 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:40.518801 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:40.526027 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:40.526124 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:10:40.526166 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:40.526227 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:40.526283 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:40.526341 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:40.535986 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39459
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:40.537273 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:40.538595 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:40.550174 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:40.550101 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:40.550143 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:40.550909 1479 server_base.cc:1034] running on GCE node
I20260928 05:10:40.551628 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:40.552932 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:40.554176 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790572240554121 us; error 48 us; skew 500 ppm
I20260928 05:10:40.554515 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:40 dist-test-slave-6718 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:40.557616 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:40.559129 1479 webserver.cc:460] Webserver started at http://127.0.97.62:40503/ using document root <none> and password file <none>
I20260928 05:10:40.559805 1479 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:40.559913 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:40.560245 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:40.563172 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "840ea87e945a4d67acadcbfac3393652"
format_stamp: "Formatted at 2026-09-28 05:10:40 on dist-test-slave-6718"
server_key: "9fcae6c3c72a45a82a144fb768b9275f"
server_key_iv: "a28ff265b97315c881fe031d4cba003e"
server_key_version: "encryptionkey@0"
I20260928 05:10:40.563964 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "840ea87e945a4d67acadcbfac3393652"
format_stamp: "Formatted at 2026-09-28 05:10:40 on dist-test-slave-6718"
server_key: "9fcae6c3c72a45a82a144fb768b9275f"
server_key_iv: "a28ff265b97315c881fe031d4cba003e"
server_key_version: "encryptionkey@0"
I20260928 05:10:40.568786 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:10:40.571985 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:40.573488 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:10:40.573665 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "840ea87e945a4d67acadcbfac3393652"
format_stamp: "Formatted at 2026-09-28 05:10:40 on dist-test-slave-6718"
server_key: "9fcae6c3c72a45a82a144fb768b9275f"
server_key_iv: "a28ff265b97315c881fe031d4cba003e"
server_key_version: "encryptionkey@0"
I20260928 05:10:40.573915 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:40.591976 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:40.595865 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:40.596186 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:40.616255 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39459
I20260928 05:10:40.616282 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39459 every 8 connection(s)
I20260928 05:10:40.617666 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:10:40.621246 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:40.627502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1479
I20260928 05:10:40.627699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:10:40.628345 388 external_mini_cluster.cc:1428] Setting key b5e0cce9ed006f82003e659d42930d75
I20260928 05:10:40.631103 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: Bootstrap starting.
I20260928 05:10:40.634387 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:40.635545 1556 log.cc:824] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:40.638507 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: No bootstrap required, opened a new log
Sep 28 05:10:40 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:40.647063 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ea87e945a4d67acadcbfac3393652" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260928 05:10:40.647392 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:40.647502 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840ea87e945a4d67acadcbfac3393652, State: Initialized, Role: FOLLOWER
I20260928 05:10:40.648092 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ea87e945a4d67acadcbfac3393652" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260928 05:10:40.648398 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:10:40.648532 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:10:40.648701 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:40.651232 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:40.630165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.0.97.62:39459)
0928 05:10:40.630911 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:40.630922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:40.630959 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:10:40.632614 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:40.632637 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:40.632994 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:40.633292 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:40.635071 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:40.636734 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:40.638430 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:40.639156 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:40.639291 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:40.643662 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:40.643670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:40.643686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:40.643721 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:40.646868 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:40.647577 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:40.647586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:40.647595 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:40.647788 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:40.648166 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:40.648171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:40.648174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:40.648887 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:40.649162 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:40.649494 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 05:10:40.649924 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":364,"threads_started":1}
I20260928 05:10:40.651849 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ea87e945a4d67acadcbfac3393652" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260928 05:10:40.652376 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840ea87e945a4d67acadcbfac3393652; no voters:
I20260928 05:10:40.653012 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:10:40.653144 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:40.653797 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [term 1 LEADER]: Becoming Leader. State: Replica: 840ea87e945a4d67acadcbfac3393652, State: Running, Role: LEADER
I20260928 05:10:40.654356 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ea87e945a4d67acadcbfac3393652" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } }
I20260928 05:10:40.655486 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:10:40.656980 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840ea87e945a4d67acadcbfac3393652. Latest consensus state: current_term: 1 leader_uuid: "840ea87e945a4d67acadcbfac3393652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ea87e945a4d67acadcbfac3393652" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } } }
I20260928 05:10:40.657295 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "840ea87e945a4d67acadcbfac3393652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840ea87e945a4d67acadcbfac3393652" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39459 } } }
I20260928 05:10:40.657641 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:40.657313 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:40.659754 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:10:40.665333 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:10:40.673666 1566 catalog_manager.cc:1347] Generated new cluster ID: f543a32397ef418d9c619b15e6981d53
I20260928 05:10:40.673787 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:10:40.712037 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:10:40.713865 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:10:40.723560 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: Generated new TSK 0
I20260928 05:10:40.724516 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:10:40.768026 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:41.085011 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:41.085305 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:41.085472 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:41.092556 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:41.092703 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:41.092854 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:10:41.092993 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:10:41.102383 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:41.103631 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:41.104943 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:41.117795 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:41.117775 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:41.117992 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:41.118930 1582 server_base.cc:1034] running on GCE node
I20260928 05:10:41.119511 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:41.120344 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:41.121562 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790572241121493 us; error 50 us; skew 500 ppm
I20260928 05:10:41.121837 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:41 dist-test-slave-6718 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:41.124897 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:10:41.126205 1582 webserver.cc:460] Webserver started at http://127.0.97.1:45523/ using document root <none> and password file <none>
I20260928 05:10:41.126922 1582 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:41.127031 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:41.127350 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:41.130060 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "a2f67074a0e443ec8d39f9d2573dfc4d"
format_stamp: "Formatted at 2026-09-28 05:10:41 on dist-test-slave-6718"
server_key: "4be1bda40e11b29d3ee12663ad51f2c3"
server_key_iv: "d4fe92e5157a3fd9b6554b05460b7459"
server_key_version: "encryptionkey@0"
I20260928 05:10:41.130879 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a2f67074a0e443ec8d39f9d2573dfc4d"
format_stamp: "Formatted at 2026-09-28 05:10:41 on dist-test-slave-6718"
server_key: "4be1bda40e11b29d3ee12663ad51f2c3"
server_key_iv: "d4fe92e5157a3fd9b6554b05460b7459"
server_key_version: "encryptionkey@0"
I20260928 05:10:41.135536 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:10:41.138796 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:41.139920 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 05:10:41.140085 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "a2f67074a0e443ec8d39f9d2573dfc4d"
format_stamp: "Formatted at 2026-09-28 05:10:41 on dist-test-slave-6718"
server_key: "4be1bda40e11b29d3ee12663ad51f2c3"
server_key_iv: "d4fe92e5157a3fd9b6554b05460b7459"
server_key_version: "encryptionkey@0"
I20260928 05:10:41.140281 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:41.150797 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:41.154335 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:41.154644 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:41.157159 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:41.157276 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:41.157424 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:41.157523 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 05:10:41 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:41.183224 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.157605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40877 (local address 127.0.97.62:39459)
0928 05:10:41.160640 (+ 3035us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:41.160645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:41.160663 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:10:41.162932 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:41.162940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:41.163038 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:41.163161 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:41.165709 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.166684 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.169861 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.170746 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.170909 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.175988 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:41.175993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:41.175997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:41.176028 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:41.179625 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.180626 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.180632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.180635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.180786 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.181257 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.181262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.181265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.181533 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:41.181676 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:41.182645 (+ 969us) server_negotiation.cc:299] Negotiation successful
0928 05:10:41.182864 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2936,"thread_start_us":2845,"threads_started":1}
I20260928 05:10:41.184859 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.158321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.1:40877)
0928 05:10:41.159396 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:41.159449 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:41.162547 (+ 3098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:41.163464 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:41.163477 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:41.164425 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:41.165487 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.165524 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.166913 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.166923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.169647 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.169669 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.171054 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.171065 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.171499 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.172226 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:41.172266 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:41.175642 (+ 3376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:41.179771 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.179780 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.179802 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.180455 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.180919 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.180924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.180928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.181139 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.181669 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:41.181680 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:41.182151 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 05:10:41.182538 (+ 387us) client_negotiation.cc:240] Negotiation successful
0928 05:10:41.183116 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":514,"threads_started":1}
I20260928 05:10:41.201116 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46859
I20260928 05:10:41.201138 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46859 every 8 connection(s)
I20260928 05:10:41.202322 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:10:41.204555 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39459
I20260928 05:10:41.204773 1718 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:41.205461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1582
I20260928 05:10:41.205585 1718 heartbeater.cc:511] Master 127.0.97.62:39459 requested a full tablet report, sending...
I20260928 05:10:41.205668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:10:41.206454 388 external_mini_cluster.cc:1428] Setting key 61cb978e243b98b714cb0c49877bd8e9
I20260928 05:10:41.206710 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.191954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60205 (local address 127.0.97.62:39459)
0928 05:10:41.192137 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:41.192146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:41.192891 (+ 745us) server_negotiation.cc:407] Connection header received
0928 05:10:41.193132 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:41.193141 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:41.193243 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:41.193381 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:41.194316 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.195627 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.198367 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.198824 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.198968 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.200952 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:41.200957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:41.200960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:41.200988 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:41.203849 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.204639 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.204647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.204650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.204799 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.205187 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.205194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.205198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.205468 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:41.205705 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:41.206084 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 05:10:41.206300 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:10:41.206760 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.191979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.1:60205)
0928 05:10:41.192797 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:41.192817 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:41.192974 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:41.193447 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:41.193454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:41.193894 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:41.194160 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.194169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.195847 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.195857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.198200 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.198220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.199021 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.199027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.199203 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.199821 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:41.199838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:41.200769 (+ 931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:41.204009 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.204013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.204017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.204512 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.204948 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.204951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.204953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.205082 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.205662 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:41.205668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:41.205852 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 05:10:41.205987 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 05:10:41.206294 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":484,"threads_started":1}
I20260928 05:10:41.208098 1511 ts_manager.cc:194] Registered new tserver with Master: a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859)
I20260928 05:10:41.210117 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40877
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:10:41.245850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:41.556808 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:41.557137 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:41.557312 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:41.564675 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:41.564787 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:41.564963 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:10:41.565099 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:10:41.574914 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:41.576161 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:41.577597 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:41.590652 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:41.590688 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:41.594105 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:41.595595 1722 server_base.cc:1034] running on GCE node
I20260928 05:10:41.596235 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:41.597167 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:41.598460 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790572241598394 us; error 44 us; skew 500 ppm
I20260928 05:10:41.598819 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:41 dist-test-slave-6718 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:41.608783 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:10:41.610201 1722 webserver.cc:460] Webserver started at http://127.0.97.2:45781/ using document root <none> and password file <none>
I20260928 05:10:41.611022 1722 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:41.611150 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:41.611480 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:41.614341 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "e0da7413c74f48cc9ba5d0e50a0780f5"
format_stamp: "Formatted at 2026-09-28 05:10:41 on dist-test-slave-6718"
server_key: "64bcd8d86a1e2d5ca91b711bf9cdc929"
server_key_iv: "55d94258c138fa2b74119fa4366495cc"
server_key_version: "encryptionkey@0"
I20260928 05:10:41.615173 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e0da7413c74f48cc9ba5d0e50a0780f5"
format_stamp: "Formatted at 2026-09-28 05:10:41 on dist-test-slave-6718"
server_key: "64bcd8d86a1e2d5ca91b711bf9cdc929"
server_key_iv: "55d94258c138fa2b74119fa4366495cc"
server_key_version: "encryptionkey@0"
I20260928 05:10:41.620251 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:10:41.623632 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:41.624985 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:10:41.625144 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "e0da7413c74f48cc9ba5d0e50a0780f5"
format_stamp: "Formatted at 2026-09-28 05:10:41 on dist-test-slave-6718"
server_key: "64bcd8d86a1e2d5ca91b711bf9cdc929"
server_key_iv: "55d94258c138fa2b74119fa4366495cc"
server_key_version: "encryptionkey@0"
I20260928 05:10:41.625365 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:41.643714 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:41.648105 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:41.648379 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:41.650892 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:41.651016 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:41.651137 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:41.651202 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:10:41 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:41.673403 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.651634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39097 (local address 127.0.97.62:39459)
0928 05:10:41.651823 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:41.651830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:41.653303 (+ 1473us) server_negotiation.cc:407] Connection header received
0928 05:10:41.655567 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:41.655575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:41.655667 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:41.655773 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:41.657992 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.659316 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.661656 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.662570 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.662734 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.666332 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:41.666341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:41.666349 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:41.666399 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:41.669636 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.670716 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.670724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.670730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.670915 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.671575 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.671582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.671585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.671970 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:41.672110 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:41.672861 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 05:10:41.673070 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:10:41.674405 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.652314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.2:39097)
0928 05:10:41.653130 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:41.653167 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:41.655274 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:41.656041 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:41.656081 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:41.656903 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:41.657770 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.657818 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.659509 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.659516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.661518 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.661530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.662841 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.662854 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.663093 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.663787 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:41.663807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:41.666106 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:41.669796 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.669805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.669815 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.670508 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.671044 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.671048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.671052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.671410 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.672186 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:41.672194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:41.672497 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 05:10:41.672784 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 05:10:41.673272 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":375,"threads_started":1}
I20260928 05:10:41.696274 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.681025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56271 (local address 127.0.97.62:39459)
0928 05:10:41.681229 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:41.681238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:41.681916 (+ 678us) server_negotiation.cc:407] Connection header received
0928 05:10:41.682152 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:41.682157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:41.682239 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:41.682358 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:41.683939 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.684842 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.686293 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.686880 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.687036 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.689657 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:41.689661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:41.689664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:41.689692 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:41.692903 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.693869 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.693878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.693884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.694033 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:41.694484 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:41.694491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:41.694495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:41.694970 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:41.695112 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:41.695702 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 05:10:41.695931 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:10:41.696435 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:41.680992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.2:56271)
0928 05:10:41.681791 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:41.681816 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:41.681989 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:41.682383 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:41.682391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:41.683187 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:41.683754 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.683781 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.684988 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.684993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.686134 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:41.686145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:41.687101 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:41.687106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:41.687390 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:41.688135 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:41.688166 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:41.689447 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:41.693039 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.693045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.693049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.693713 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.694168 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:41.694175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:41.694180 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:41.694359 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:41.695078 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:41.695083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:41.695373 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 05:10:41.695606 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 05:10:41.695948 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":393,"threads_started":1}
I20260928 05:10:41.708137 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42793
I20260928 05:10:41.708158 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42793 every 8 connection(s)
I20260928 05:10:41.709321 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:10:41.711324 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39459
I20260928 05:10:41.711547 1857 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:41.712158 1857 heartbeater.cc:511] Master 127.0.97.62:39459 requested a full tablet report, sending...
I20260928 05:10:41.712280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1722
I20260928 05:10:41.712461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:10:41.713070 388 external_mini_cluster.cc:1428] Setting key 4e96f2f24034077683315b31d3e7e303
I20260928 05:10:41.713537 1510 ts_manager.cc:194] Registered new tserver with Master: e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793)
I20260928 05:10:41.714816 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39097
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:10:41.751847 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39459
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:42.066254 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:42.066550 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:42.066649 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:42.073863 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:42.074014 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:42.074169 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:10:42.074311 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:10:42.084254 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:42.085567 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:42.087014 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:42.100131 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:42.100291 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:42.100054 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:42.100929 1861 server_base.cc:1034] running on GCE node
I20260928 05:10:42.101552 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:42.102484 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:42.103742 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790572242103689 us; error 37 us; skew 500 ppm
I20260928 05:10:42.104046 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:42 dist-test-slave-6718 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:42.107630 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:10:42.109680 1861 webserver.cc:460] Webserver started at http://127.0.97.3:32861/ using document root <none> and password file <none>
I20260928 05:10:42.110628 1861 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:42.110739 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:42.111130 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:42.114425 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "28845aca0f28479dbd5624bfee6c5836"
format_stamp: "Formatted at 2026-09-28 05:10:42 on dist-test-slave-6718"
server_key: "20667bb7bdc5886da851ffc94f023564"
server_key_iv: "63976d30a2a76b2fe68c4886502f8165"
server_key_version: "encryptionkey@0"
I20260928 05:10:42.115268 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "28845aca0f28479dbd5624bfee6c5836"
format_stamp: "Formatted at 2026-09-28 05:10:42 on dist-test-slave-6718"
server_key: "20667bb7bdc5886da851ffc94f023564"
server_key_iv: "63976d30a2a76b2fe68c4886502f8165"
server_key_version: "encryptionkey@0"
I20260928 05:10:42.120141 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:10:42.123423 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:42.124770 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:10:42.124933 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "28845aca0f28479dbd5624bfee6c5836"
format_stamp: "Formatted at 2026-09-28 05:10:42 on dist-test-slave-6718"
server_key: "20667bb7bdc5886da851ffc94f023564"
server_key_iv: "63976d30a2a76b2fe68c4886502f8165"
server_key_version: "encryptionkey@0"
I20260928 05:10:42.125129 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:42.162854 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:42.166926 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:42.167303 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:42.169970 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:42.170054 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:42.170182 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:42.170271 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:10:42 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:42.196615 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.170496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35535 (local address 127.0.97.62:39459)
0928 05:10:42.170724 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.170730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.172387 (+ 1657us) server_negotiation.cc:407] Connection header received
0928 05:10:42.175360 (+ 2973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.175366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.175455 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.175552 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:42.177881 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.179175 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.181978 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.182787 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.182906 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.188923 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:42.188930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:42.188935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:42.188978 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:42.192270 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:42.193474 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:42.193482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:42.193488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:42.193650 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:42.194334 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:42.194338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:42.194340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:42.194740 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:42.194897 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.195984 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.196212 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 05:10:42.197705 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.171113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.3:35535)
0928 05:10:42.172140 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.172184 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.175049 (+ 2865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.175959 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.175972 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.176758 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:42.177640 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.177685 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.179433 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.179441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.181808 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.181825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.183328 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.183336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.183682 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.184648 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:42.184688 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:42.188697 (+ 4009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:42.192440 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:42.192457 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:42.192488 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:42.193318 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:42.193791 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:42.193797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:42.193800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:42.194163 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:42.194968 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:42.194975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:42.195437 (+ 462us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.195852 (+ 415us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.196427 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":390,"threads_started":1}
I20260928 05:10:42.213671 1718 heartbeater.cc:503] Master 127.0.97.62:39459 was elected leader, sending a full tablet report...
I20260928 05:10:42.224125 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.209731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37483 (local address 127.0.97.62:39459)
0928 05:10:42.209948 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.209957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.211422 (+ 1465us) server_negotiation.cc:407] Connection header received
0928 05:10:42.211655 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.211662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.211798 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.211926 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:42.213398 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.214479 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.216049 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.216499 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.216637 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.218377 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:42.218383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:42.218388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:42.218431 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:42.221176 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:42.221846 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:42.221852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:42.221856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:42.221977 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:42.222327 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:42.222334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:42.222338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:42.222683 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:42.222846 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.223517 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.223739 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:10:42.224226 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.209848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.97.3:37483)
0928 05:10:42.211217 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.211243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.211509 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.211946 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.211951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.212668 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:42.213243 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.213260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.214614 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.214621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.215913 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.215923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.216676 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.216682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.216857 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.217477 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:42.217494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:42.218196 (+ 702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:42.221316 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:42.221323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:42.221327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:42.221727 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:42.222083 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:42.222088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:42.222092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:42.222217 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:42.222957 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:42.222965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:42.223213 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.223416 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.223764 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":631,"threads_started":1}
I20260928 05:10:42.239020 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42945
I20260928 05:10:42.239042 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42945 every 8 connection(s)
I20260928 05:10:42.240209 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:10:42.242249 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39459
I20260928 05:10:42.242475 1996 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:42.243124 1996 heartbeater.cc:511] Master 127.0.97.62:39459 requested a full tablet report, sending...
I20260928 05:10:42.244426 1510 ts_manager.cc:194] Registered new tserver with Master: 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945)
I20260928 05:10:42.245352 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35535
I20260928 05:10:42.250063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 1861
I20260928 05:10:42.250240 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:10:42.250916 388 external_mini_cluster.cc:1428] Setting key 0a4c519d97efa247827bd5e365281f4e
I20260928 05:10:42.253832 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:10:42 dist-test-slave-6718 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:10:42 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:42.298240 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.284382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.97.62:39459)
0928 05:10:42.284617 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.284623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.284852 (+ 229us) server_negotiation.cc:407] Connection header received
0928 05:10:42.285058 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.285069 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.285127 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.285229 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:42.286613 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.287490 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.288815 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.289222 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.289359 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.293028 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:42.293033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:42.293036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:42.293068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:42.295735 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:42.296348 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:42.296355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:42.296359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:42.296491 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:42.296859 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:42.296866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:42.296869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:42.297121 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:42.297362 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.297670 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.297904 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:10:42.302218 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:10:42.308781 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:42.331015 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.320742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.0.97.2:42793)
0928 05:10:42.322939 (+ 2197us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.322952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.322987 (+ 35us) server_negotiation.cc:407] Connection header received
0928 05:10:42.323109 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.323120 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.323424 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.323670 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.324879 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.326131 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.328809 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.329923 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.330093 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.330296 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.330563 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.330722 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":1789,"threads_started":1}
I20260928 05:10:42.331090 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.320592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42793 (local address 127.0.0.1:57132)
0928 05:10:42.321385 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.321430 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.321601 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.323695 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.323707 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.323770 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.324364 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.324381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.326262 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.326267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.328573 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.328595 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.330103 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.330108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.330270 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.330335 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.330489 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.330695 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":2,"thread_start_us":390,"threads_started":1}
I20260928 05:10:42.332608 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.321666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.0.97.3:42945)
0928 05:10:42.324489 (+ 2823us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.324499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.324530 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:10:42.324615 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.324622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.324869 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.325604 (+ 735us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.325912 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.327582 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.330114 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.331579 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.331707 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.331940 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.332109 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.332269 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2662,"thread_start_us":2481,"threads_started":1}
I20260928 05:10:42.333236 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.322569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.0.97.1:46859)
0928 05:10:42.323177 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.323189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.325135 (+ 1946us) server_negotiation.cc:407] Connection header received
0928 05:10:42.325223 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.325230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.325423 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.325591 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.326243 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.327259 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.330074 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.331845 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.331971 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.332160 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.332848 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.332982 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":323,"threads_started":1}
I20260928 05:10:42.333796 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.321019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42945 (local address 127.0.0.1:39936)
0928 05:10:42.322241 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.322258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.322358 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.325068 (+ 2710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.325072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.325093 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.325487 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.325500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.327720 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.327727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.329949 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.329964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.331772 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.331785 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.331939 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.331969 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.332061 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.332189 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":948,"threads_started":1}
I20260928 05:10:42.333859 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.321982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46859 (local address 127.0.0.1:41642)
0928 05:10:42.324899 (+ 2917us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.324916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.325025 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.325647 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.325654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.325679 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.326059 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.326070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.327400 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.327405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.329786 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.329801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.332315 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.332322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.332489 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.332518 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.332609 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.332750 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2807,"thread_start_us":2675,"threads_started":1}
I20260928 05:10:42.335170 1787 tablet_service.cc:1467] Processing CreateTablet for tablet d0cc42eb4a2541219689282611061f41 (DEFAULT_TABLE table=test-table [id=555d091686d54e7bb3649546ba9e1163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:42.336609 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cc42eb4a2541219689282611061f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:42.336870 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d0cc42eb4a2541219689282611061f41 (DEFAULT_TABLE table=test-table [id=555d091686d54e7bb3649546ba9e1163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:42.337235 1926 tablet_service.cc:1467] Processing CreateTablet for tablet d0cc42eb4a2541219689282611061f41 (DEFAULT_TABLE table=test-table [id=555d091686d54e7bb3649546ba9e1163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:42.337973 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cc42eb4a2541219689282611061f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:42.338483 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cc42eb4a2541219689282611061f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:42.348889 2015 tablet_bootstrap.cc:492] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Bootstrap starting.
I20260928 05:10:42.351464 2016 tablet_bootstrap.cc:492] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Bootstrap starting.
I20260928 05:10:42.352937 2015 tablet_bootstrap.cc:654] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:42.354176 2015 log.cc:824] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:42.354640 2016 tablet_bootstrap.cc:654] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:42.355096 2017 tablet_bootstrap.cc:492] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Bootstrap starting.
I20260928 05:10:42.355980 2016 log.cc:824] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:42.358750 2016 tablet_bootstrap.cc:492] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: No bootstrap required, opened a new log
I20260928 05:10:42.358752 2015 tablet_bootstrap.cc:492] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: No bootstrap required, opened a new log
I20260928 05:10:42.358742 2017 tablet_bootstrap.cc:654] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:42.359014 2016 ts_tablet_manager.cc:1397] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260928 05:10:42.359033 2015 ts_tablet_manager.cc:1397] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 05:10:42.359892 2017 log.cc:824] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:42.363248 2017 tablet_bootstrap.cc:492] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: No bootstrap required, opened a new log
I20260928 05:10:42.363500 2017 ts_tablet_manager.cc:1397] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 05:10:42.366923 2016 raft_consensus.cc:348] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.367295 2016 raft_consensus.cc:374] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:42.367417 2016 raft_consensus.cc:729] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0da7413c74f48cc9ba5d0e50a0780f5, State: Initialized, Role: FOLLOWER
I20260928 05:10:42.368012 2016 consensus_queue.cc:260] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.369057 1857 heartbeater.cc:503] Master 127.0.97.62:39459 was elected leader, sending a full tablet report...
I20260928 05:10:42.369563 2016 ts_tablet_manager.cc:1428] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260928 05:10:42.370146 2015 raft_consensus.cc:348] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.370595 2015 raft_consensus.cc:374] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:42.370697 2015 raft_consensus.cc:729] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f67074a0e443ec8d39f9d2573dfc4d, State: Initialized, Role: FOLLOWER
I20260928 05:10:42.371294 2015 consensus_queue.cc:260] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.372926 2015 ts_tablet_manager.cc:1428] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 05:10:42.375429 2017 raft_consensus.cc:348] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.375876 2017 raft_consensus.cc:374] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:42.375975 2017 raft_consensus.cc:729] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28845aca0f28479dbd5624bfee6c5836, State: Initialized, Role: FOLLOWER
I20260928 05:10:42.376597 2017 consensus_queue.cc:260] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.377688 1996 heartbeater.cc:503] Master 127.0.97.62:39459 was elected leader, sending a full tablet report...
I20260928 05:10:42.378302 2017 ts_tablet_manager.cc:1428] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20260928 05:10:42.441615 2022 raft_consensus.cc:482] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:42.442291 2022 raft_consensus.cc:504] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.443840 2022 leader_election.cc:288] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793), 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945)
I20260928 05:10:42.452126 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.444221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48247 (local address 127.0.97.2:42793)
0928 05:10:42.444409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.444416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.445187 (+ 771us) server_negotiation.cc:407] Connection header received
0928 05:10:42.445349 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.445353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.445409 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.445486 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.446021 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.447096 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.449779 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.451022 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.451147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.451248 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.451741 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.451868 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:10:42.452123 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.444229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33673 (local address 127.0.97.3:42945)
0928 05:10:42.444417 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.444424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.445143 (+ 719us) server_negotiation.cc:407] Connection header received
0928 05:10:42.445623 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.445626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.445677 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.445740 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.446318 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.447326 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.449793 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.451022 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.451147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.451248 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.451741 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.451879 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 05:10:42.452348 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.444138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42945 (local address 127.0.97.1:33673)
0928 05:10:42.445023 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.445044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.445211 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.445793 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.445799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.445826 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.446181 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.446189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.447432 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.447437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.449613 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.449632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.451297 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.451301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.451546 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.451604 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.451699 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.451870 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":132,"spinlock_wait_cycles":28288,"thread_start_us":517,"threads_started":1}
I20260928 05:10:42.452965 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.444179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42793 (local address 127.0.97.1:48247)
0928 05:10:42.445019 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.445040 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.445211 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.445520 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.445523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.445543 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.445856 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.445871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.447217 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.447220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.449629 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.449638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.451287 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.451293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.451541 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.451611 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.451699 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.451870 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":142,"spinlock_wait_cycles":14720,"thread_start_us":450,"threads_started":1}
I20260928 05:10:42.453320 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cc42eb4a2541219689282611061f41" candidate_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28845aca0f28479dbd5624bfee6c5836" is_pre_election: true
I20260928 05:10:42.454344 1951 raft_consensus.cc:2454] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f67074a0e443ec8d39f9d2573dfc4d in term 0.
I20260928 05:10:42.454319 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cc42eb4a2541219689282611061f41" candidate_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" is_pre_election: true
W20260928 05:10:42.454756 1719 tablet.cc:2369] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:10:42.454851 1812 raft_consensus.cc:2454] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f67074a0e443ec8d39f9d2573dfc4d in term 0.
I20260928 05:10:42.455098 1599 leader_election.cc:302] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28845aca0f28479dbd5624bfee6c5836, a2f67074a0e443ec8d39f9d2573dfc4d; no voters:
I20260928 05:10:42.455579 2022 raft_consensus.cc:2781] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:42.455736 2022 raft_consensus.cc:482] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:42.455856 2022 raft_consensus.cc:3037] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:42.457993 2022 raft_consensus.cc:504] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.458585 2022 leader_election.cc:288] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [CANDIDATE]: Term 1 election: Requested vote from peers e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793), 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945)
I20260928 05:10:42.459098 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cc42eb4a2541219689282611061f41" candidate_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28845aca0f28479dbd5624bfee6c5836"
I20260928 05:10:42.459098 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cc42eb4a2541219689282611061f41" candidate_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5"
I20260928 05:10:42.459311 1951 raft_consensus.cc:3037] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:42.459301 1812 raft_consensus.cc:3037] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Advancing to term 1
W20260928 05:10:42.461107 1858 tablet.cc:2369] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:10:42.462596 1951 raft_consensus.cc:2454] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f67074a0e443ec8d39f9d2573dfc4d in term 1.
I20260928 05:10:42.462596 1812 raft_consensus.cc:2454] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f67074a0e443ec8d39f9d2573dfc4d in term 1.
I20260928 05:10:42.463024 1599 leader_election.cc:302] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28845aca0f28479dbd5624bfee6c5836, a2f67074a0e443ec8d39f9d2573dfc4d; no voters:
I20260928 05:10:42.463300 2022 raft_consensus.cc:2781] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:42.463714 2022 raft_consensus.cc:686] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 LEADER]: Becoming Leader. State: Replica: a2f67074a0e443ec8d39f9d2573dfc4d, State: Running, Role: LEADER
I20260928 05:10:42.464169 2022 consensus_queue.cc:237] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.469043 1510 catalog_manager.cc:5495] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } health_report { overall_health: UNKNOWN } } }
W20260928 05:10:42.491716 1997 tablet.cc:2369] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:10:42.495386 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.490041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.97.1:46859)
0928 05:10:42.490241 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.490247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.490387 (+ 140us) server_negotiation.cc:407] Connection header received
0928 05:10:42.490677 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.490681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.490781 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.490895 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:42.491197 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.491920 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.493352 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.493718 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.493815 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.494228 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:42.494776 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:42.494878 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.495030 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.495089 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:10:42.503063 1951 raft_consensus.cc:1261] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Refusing update from remote peer a2f67074a0e443ec8d39f9d2573dfc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:42.503072 1812 raft_consensus.cc:1261] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Refusing update from remote peer a2f67074a0e443ec8d39f9d2573dfc4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:42.504144 2026 consensus_queue.cc:1035] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:42.504652 2022 consensus_queue.cc:1035] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:42.518024 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334183905282371584 to 7334183905136656384. Current Snapshot: MvccSnapshot[applied={T|T < 7334183905282371584}]
I20260928 05:10:42.532383 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:42.532626 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:42.536410 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: Sending DeleteTablet for 3 replicas of tablet d0cc42eb4a2541219689282611061f41
I20260928 05:10:42.537545 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cc42eb4a2541219689282611061f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41642
I20260928 05:10:42.537772 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cc42eb4a2541219689282611061f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57132
I20260928 05:10:42.538723 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cc42eb4a2541219689282611061f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39936
I20260928 05:10:42.539269 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:10:42.540086 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:42.547437 2040 tablet_replica.cc:330] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: stopping tablet replica
I20260928 05:10:42.547878 2040 raft_consensus.cc:2229] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:42.548363 2040 raft_consensus.cc:2258] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:42.550433 1647 tablet_service.cc:1467] Processing CreateTablet for tablet c9fd45388e3b4bb5aa226bda31d2ee56 (DEFAULT_TABLE table=test-table [id=601a7aee72434eeba0e38a1424cbc0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:42.550470 1786 tablet_service.cc:1467] Processing CreateTablet for tablet c9fd45388e3b4bb5aa226bda31d2ee56 (DEFAULT_TABLE table=test-table [id=601a7aee72434eeba0e38a1424cbc0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:42.551023 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fd45388e3b4bb5aa226bda31d2ee56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:42.551043 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fd45388e3b4bb5aa226bda31d2ee56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:42.551024 1925 tablet_service.cc:1467] Processing CreateTablet for tablet c9fd45388e3b4bb5aa226bda31d2ee56 (DEFAULT_TABLE table=test-table [id=601a7aee72434eeba0e38a1424cbc0b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:42.551549 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fd45388e3b4bb5aa226bda31d2ee56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:42.558058 2016 tablet_bootstrap.cc:492] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Bootstrap starting.
I20260928 05:10:42.558081 2015 tablet_bootstrap.cc:492] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Bootstrap starting.
I20260928 05:10:42.560097 2016 tablet_bootstrap.cc:654] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:42.560101 2015 tablet_bootstrap.cc:654] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:42.560312 2017 tablet_bootstrap.cc:492] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Bootstrap starting.
I20260928 05:10:42.560467 2041 tablet_replica.cc:330] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: stopping tablet replica
I20260928 05:10:42.560964 2041 raft_consensus.cc:2229] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:42.561687 2041 raft_consensus.cc:2258] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:42.562299 2017 tablet_bootstrap.cc:654] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:42.564481 2041 ts_tablet_manager.cc:1905] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:42.571869 2041 ts_tablet_manager.cc:1918] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:10:42.572104 2041 log.cc:1196] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/d0cc42eb4a2541219689282611061f41
I20260928 05:10:42.572554 2017 tablet_bootstrap.cc:492] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: No bootstrap required, opened a new log
I20260928 05:10:42.572701 2017 ts_tablet_manager.cc:1397] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260928 05:10:42.572898 2041 ts_tablet_manager.cc:1939] T d0cc42eb4a2541219689282611061f41 P 28845aca0f28479dbd5624bfee6c5836: Deleting consensus metadata
I20260928 05:10:42.573541 2016 tablet_bootstrap.cc:492] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: No bootstrap required, opened a new log
I20260928 05:10:42.573694 2016 ts_tablet_manager.cc:1397] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20260928 05:10:42.573597 2017 raft_consensus.cc:348] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.573800 2017 raft_consensus.cc:374] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:42.573909 2017 raft_consensus.cc:729] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28845aca0f28479dbd5624bfee6c5836, State: Initialized, Role: FOLLOWER
I20260928 05:10:42.574151 2017 consensus_queue.cc:260] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.574434 2016 raft_consensus.cc:348] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.574674 2016 raft_consensus.cc:374] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:42.574795 2016 raft_consensus.cc:729] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0da7413c74f48cc9ba5d0e50a0780f5, State: Initialized, Role: FOLLOWER
I20260928 05:10:42.575037 2016 consensus_queue.cc:260] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.575536 2016 ts_tablet_manager.cc:1428] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:10:42.575629 1496 catalog_manager.cc:4841] TS 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945): tablet d0cc42eb4a2541219689282611061f41 (table test-table [id=555d091686d54e7bb3649546ba9e1163]) successfully deleted
I20260928 05:10:42.577936 2015 tablet_bootstrap.cc:492] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: No bootstrap required, opened a new log
I20260928 05:10:42.578096 2015 ts_tablet_manager.cc:1397] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.004s
I20260928 05:10:42.578104 2017 ts_tablet_manager.cc:1428] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260928 05:10:42.578951 2015 raft_consensus.cc:348] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.579173 2015 raft_consensus.cc:374] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:42.579279 2015 raft_consensus.cc:729] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f67074a0e443ec8d39f9d2573dfc4d, State: Initialized, Role: FOLLOWER
I20260928 05:10:42.579977 2015 consensus_queue.cc:260] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.580812 2015 ts_tablet_manager.cc:1428] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 05:10:42.581308 2039 tablet_replica.cc:330] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: stopping tablet replica
I20260928 05:10:42.581807 2039 raft_consensus.cc:2229] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:10:42.582233 2040 ts_tablet_manager.cc:1905] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:42.582453 2039 raft_consensus.cc:2258] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:42.584316 2039 ts_tablet_manager.cc:1905] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:42.589664 2040 ts_tablet_manager.cc:1918] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:10:42.589820 2040 log.cc:1196] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/d0cc42eb4a2541219689282611061f41
I20260928 05:10:42.590829 2040 ts_tablet_manager.cc:1939] T d0cc42eb4a2541219689282611061f41 P e0da7413c74f48cc9ba5d0e50a0780f5: Deleting consensus metadata
I20260928 05:10:42.591871 2039 ts_tablet_manager.cc:1918] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:10:42.592024 2039 log.cc:1196] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/d0cc42eb4a2541219689282611061f41
I20260928 05:10:42.592554 2039 ts_tablet_manager.cc:1939] T d0cc42eb4a2541219689282611061f41 P a2f67074a0e443ec8d39f9d2573dfc4d: Deleting consensus metadata
I20260928 05:10:42.592547 1495 catalog_manager.cc:4841] TS e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793): tablet d0cc42eb4a2541219689282611061f41 (table test-table [id=555d091686d54e7bb3649546ba9e1163]) successfully deleted
I20260928 05:10:42.593794 1495 catalog_manager.cc:4841] TS a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859): tablet d0cc42eb4a2541219689282611061f41 (table test-table [id=555d091686d54e7bb3649546ba9e1163]) successfully deleted
I20260928 05:10:42.914354 2023 raft_consensus.cc:482] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:42.914598 2023 raft_consensus.cc:504] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.916199 2023 leader_election.cc:288] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859), e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793)
I20260928 05:10:42.922987 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.916503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46859 (local address 127.0.97.3:37779)
0928 05:10:42.917009 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.917024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.917192 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.917814 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.917818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.917851 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.918097 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.918110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.919420 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.919428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.920737 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.920750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.922196 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.922202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.922391 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.922424 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.922516 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.922666 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":292,"threads_started":1}
I20260928 05:10:42.923019 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.916620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37779 (local address 127.0.97.1:46859)
0928 05:10:42.916816 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.916824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.917119 (+ 295us) server_negotiation.cc:407] Connection header received
0928 05:10:42.917365 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.917373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.917465 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.917539 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.918261 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.919281 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.920909 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.922017 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.922156 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.922253 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.922597 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.922718 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:10:42.924280 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.917184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42793 (local address 127.0.97.3:54713)
0928 05:10:42.917870 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:42.917890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:42.917995 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:42.918376 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:42.918380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:42.918406 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:42.918696 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.918706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.920498 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.920504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.921825 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:42.921836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.923506 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.923511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:42.923720 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.923755 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:10:42.923855 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 05:10:42.923987 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":467,"threads_started":1}
I20260928 05:10:42.924342 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:42.917450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54713 (local address 127.0.97.2:42793)
0928 05:10:42.917629 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:42.917636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:42.917981 (+ 345us) server_negotiation.cc:407] Connection header received
0928 05:10:42.918138 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:42.918144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:42.918234 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:42.918343 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:42.918880 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.920356 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.921976 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:42.923158 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:42.923270 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:42.923353 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:42.923917 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 05:10:42.924047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:10:42.924250 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fd45388e3b4bb5aa226bda31d2ee56" candidate_uuid: "28845aca0f28479dbd5624bfee6c5836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" is_pre_election: true
I20260928 05:10:42.924741 1668 raft_consensus.cc:2454] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28845aca0f28479dbd5624bfee6c5836 in term 0.
I20260928 05:10:42.925063 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fd45388e3b4bb5aa226bda31d2ee56" candidate_uuid: "28845aca0f28479dbd5624bfee6c5836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" is_pre_election: true
I20260928 05:10:42.925305 1812 raft_consensus.cc:2454] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28845aca0f28479dbd5624bfee6c5836 in term 0.
I20260928 05:10:42.925448 1877 leader_election.cc:302] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28845aca0f28479dbd5624bfee6c5836, a2f67074a0e443ec8d39f9d2573dfc4d; no voters:
I20260928 05:10:42.925851 2023 raft_consensus.cc:2781] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:42.925976 2023 raft_consensus.cc:482] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:42.926079 2023 raft_consensus.cc:3037] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:42.928334 2023 raft_consensus.cc:504] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.928949 2023 leader_election.cc:288] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [CANDIDATE]: Term 1 election: Requested vote from peers a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859), e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793)
I20260928 05:10:42.929373 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fd45388e3b4bb5aa226bda31d2ee56" candidate_uuid: "28845aca0f28479dbd5624bfee6c5836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d"
I20260928 05:10:42.929594 1668 raft_consensus.cc:3037] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:42.929560 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fd45388e3b4bb5aa226bda31d2ee56" candidate_uuid: "28845aca0f28479dbd5624bfee6c5836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5"
I20260928 05:10:42.929750 1812 raft_consensus.cc:3037] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:42.932750 1812 raft_consensus.cc:2454] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28845aca0f28479dbd5624bfee6c5836 in term 1.
I20260928 05:10:42.932760 1668 raft_consensus.cc:2454] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28845aca0f28479dbd5624bfee6c5836 in term 1.
I20260928 05:10:42.933121 1877 leader_election.cc:302] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28845aca0f28479dbd5624bfee6c5836, a2f67074a0e443ec8d39f9d2573dfc4d; no voters:
I20260928 05:10:42.933415 2023 raft_consensus.cc:2781] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:42.934379 2023 raft_consensus.cc:686] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 1 LEADER]: Becoming Leader. State: Replica: 28845aca0f28479dbd5624bfee6c5836, State: Running, Role: LEADER
I20260928 05:10:42.934842 2023 consensus_queue.cc:237] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:42.938053 1511 catalog_manager.cc:5495] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "28845aca0f28479dbd5624bfee6c5836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } health_report { overall_health: HEALTHY } } }
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:43.069566 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.056900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39459 (local address 127.0.0.1:40684)
0928 05:10:43.057609 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:43.057627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:43.057776 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:43.058080 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:43.058086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:43.058749 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:43.058947 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.058955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.059937 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.059940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.060911 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.060922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.061649 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.061654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.061792 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.062404 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:43.062420 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:43.064489 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:43.067175 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:43.067184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:43.067189 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:43.067571 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:43.067874 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:43.067877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:43.067879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:43.067994 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:43.068399 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:43.068404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:43.068532 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 05:10:43.068654 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 05:10:43.068863 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":411,"threads_started":1}
I20260928 05:10:43.069591 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.056874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.97.62:39459)
0928 05:10:43.057679 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.057683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.057705 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:10:43.057879 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.057883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.057935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.058031 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:43.059076 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.059826 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.061042 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.061485 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.061576 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.064648 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:43.064652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:43.064655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:43.064680 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:43.067062 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.067663 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.067667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.067669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.067776 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.068076 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.068079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.068080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.068301 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:43.068401 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.068743 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.068970 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":679,"thread_start_us":438,"threads_started":1}
I20260928 05:10:43.074353 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:10:43.075392 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:43.081936 1648 tablet_service.cc:1467] Processing CreateTablet for tablet e2c4aa9d0a3341a39efa545ae4978281 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba5942df504d42bda04a84d818194a4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:10:43.082119 1786 tablet_service.cc:1467] Processing CreateTablet for tablet e2c4aa9d0a3341a39efa545ae4978281 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba5942df504d42bda04a84d818194a4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:10:43.082340 1925 tablet_service.cc:1467] Processing CreateTablet for tablet e2c4aa9d0a3341a39efa545ae4978281 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba5942df504d42bda04a84d818194a4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:10:43.082509 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c4aa9d0a3341a39efa545ae4978281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:43.082649 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c4aa9d0a3341a39efa545ae4978281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:43.082832 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2c4aa9d0a3341a39efa545ae4978281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:43.088140 2056 tablet_bootstrap.cc:492] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: Bootstrap starting.
I20260928 05:10:43.089529 2056 tablet_bootstrap.cc:654] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:43.091120 2057 tablet_bootstrap.cc:492] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: Bootstrap starting.
I20260928 05:10:43.091614 2058 tablet_bootstrap.cc:492] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: Bootstrap starting.
I20260928 05:10:43.091732 2056 tablet_bootstrap.cc:492] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: No bootstrap required, opened a new log
I20260928 05:10:43.091866 2056 ts_tablet_manager.cc:1397] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:10:43.092671 2056 raft_consensus.cc:348] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.092891 2056 raft_consensus.cc:374] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:43.092985 2056 raft_consensus.cc:729] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f67074a0e443ec8d39f9d2573dfc4d, State: Initialized, Role: FOLLOWER
I20260928 05:10:43.093320 2058 tablet_bootstrap.cc:654] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:43.093200 2056 consensus_queue.cc:260] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.094332 2056 ts_tablet_manager.cc:1428] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 05:10:43.094488 2060 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.095242 2060 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.096074 2057 tablet_bootstrap.cc:654] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:43.097941 2058 tablet_bootstrap.cc:492] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: No bootstrap required, opened a new log
I20260928 05:10:43.098029 2057 tablet_bootstrap.cc:492] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: No bootstrap required, opened a new log
I20260928 05:10:43.098094 2058 ts_tablet_manager.cc:1397] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 05:10:43.098307 2057 ts_tablet_manager.cc:1397] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 05:10:43.098872 2058 raft_consensus.cc:348] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.099113 2058 raft_consensus.cc:374] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:43.099057 2057 raft_consensus.cc:348] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.099201 2058 raft_consensus.cc:729] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0da7413c74f48cc9ba5d0e50a0780f5, State: Initialized, Role: FOLLOWER
I20260928 05:10:43.099289 2057 raft_consensus.cc:374] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:43.099381 2057 raft_consensus.cc:729] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28845aca0f28479dbd5624bfee6c5836, State: Initialized, Role: FOLLOWER
I20260928 05:10:43.099452 2058 consensus_queue.cc:260] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.099584 2057 consensus_queue.cc:260] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.100085 2023 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.100440 2023 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.101001 2057 ts_tablet_manager.cc:1428] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:10:43.101737 2058 ts_tablet_manager.cc:1428] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 05:10:43.101567 2063 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.101930 2063 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.346881 2063 raft_consensus.cc:482] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:43.347082 2063 raft_consensus.cc:504] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.348431 2063 leader_election.cc:288] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859), 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945)
I20260928 05:10:43.357368 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.348703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55005 (local address 127.0.97.1:46859)
0928 05:10:43.348961 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.348970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.349914 (+ 944us) server_negotiation.cc:407] Connection header received
0928 05:10:43.350074 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.350080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.350156 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.350253 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:43.351044 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.352172 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.354996 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.356268 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.356381 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.356474 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.356960 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.357090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 05:10:43.357436 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.348903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58657 (local address 127.0.97.3:42945)
0928 05:10:43.350041 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.350046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.350062 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:10:43.350119 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.350124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.350198 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.350306 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:43.351048 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.352181 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.354996 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.356254 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.356381 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.356482 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.356960 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.357090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":1068,"threads_started":1}
I20260928 05:10:43.357532 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.348663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46859 (local address 127.0.97.2:55005)
0928 05:10:43.349796 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:43.349814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:43.349932 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:43.350313 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:43.350318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:43.350344 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:43.350879 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.350887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.352306 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.352311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.354850 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.354858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.356515 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.356520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.356777 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.356832 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:10:43.356922 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:10:43.357090 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":357,"thread_start_us":755,"threads_started":1}
I20260928 05:10:43.357981 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.348661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42945 (local address 127.0.97.2:58657)
0928 05:10:43.349481 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:43.349503 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:43.349648 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:43.350327 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:43.350330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:43.350366 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:43.350874 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.350887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.352305 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.352311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.354843 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.354858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.356487 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.356499 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.356760 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.356827 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 05:10:43.356922 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:10:43.357090 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":210,"spinlock_wait_cycles":6528,"thread_start_us":436,"threads_started":1}
I20260928 05:10:43.358134 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c4aa9d0a3341a39efa545ae4978281" candidate_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" is_pre_election: true
I20260928 05:10:43.358421 1668 raft_consensus.cc:2454] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0da7413c74f48cc9ba5d0e50a0780f5 in term 0.
I20260928 05:10:43.358696 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c4aa9d0a3341a39efa545ae4978281" candidate_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28845aca0f28479dbd5624bfee6c5836" is_pre_election: true
I20260928 05:10:43.358958 1951 raft_consensus.cc:2454] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0da7413c74f48cc9ba5d0e50a0780f5 in term 0.
I20260928 05:10:43.358932 1738 leader_election.cc:302] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f67074a0e443ec8d39f9d2573dfc4d, e0da7413c74f48cc9ba5d0e50a0780f5; no voters:
I20260928 05:10:43.359328 2063 raft_consensus.cc:2781] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:43.359462 2063 raft_consensus.cc:482] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:43.359558 2063 raft_consensus.cc:3037] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:43.362102 2063 raft_consensus.cc:504] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.362705 2063 leader_election.cc:288] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [CANDIDATE]: Term 1 election: Requested vote from peers a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859), 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945)
I20260928 05:10:43.363060 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c4aa9d0a3341a39efa545ae4978281" candidate_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d"
I20260928 05:10:43.363186 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2c4aa9d0a3341a39efa545ae4978281" candidate_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28845aca0f28479dbd5624bfee6c5836"
I20260928 05:10:43.363284 1668 raft_consensus.cc:3037] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:43.363385 1951 raft_consensus.cc:3037] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:43.366156 1668 raft_consensus.cc:2454] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0da7413c74f48cc9ba5d0e50a0780f5 in term 1.
I20260928 05:10:43.366174 1951 raft_consensus.cc:2454] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0da7413c74f48cc9ba5d0e50a0780f5 in term 1.
I20260928 05:10:43.366580 1738 leader_election.cc:302] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28845aca0f28479dbd5624bfee6c5836, a2f67074a0e443ec8d39f9d2573dfc4d, e0da7413c74f48cc9ba5d0e50a0780f5; no voters:
I20260928 05:10:43.366914 2063 raft_consensus.cc:2781] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:43.367336 2063 raft_consensus.cc:686] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 LEADER]: Becoming Leader. State: Replica: e0da7413c74f48cc9ba5d0e50a0780f5, State: Running, Role: LEADER
I20260928 05:10:43.367702 2063 consensus_queue.cc:237] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } }
I20260928 05:10:43.369488 2067 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: TxnStatusTablet state changed. Reason: New leader e0da7413c74f48cc9ba5d0e50a0780f5. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.369901 2067 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:10:43.370592 2069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:10:43.370827 2069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:10:43.371244 1511 catalog_manager.cc:5495] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } health_report { overall_health: UNKNOWN } } }
I20260928 05:10:43.391578 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.385155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42793 (local address 127.0.0.1:57136)
0928 05:10:43.385399 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:43.385418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:43.385554 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:43.385979 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:43.385983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:43.385997 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:43.386163 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.386172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.387757 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.387763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.389096 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.389108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.390069 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.390073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.390201 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.390251 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.391196 (+ 945us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.391210 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 05:10:43.391263 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 05:10:43.391369 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 05:10:43.391681 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.385257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.97.2:42793)
0928 05:10:43.385417 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.385423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.385473 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:10:43.385734 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.385744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.385819 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.385949 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:43.386306 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.387574 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.389225 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.389916 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.390067 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.390419 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.391097 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.391247 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.391304 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.391395 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:10:43.395846 1951 raft_consensus.cc:1261] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Refusing update from remote peer e0da7413c74f48cc9ba5d0e50a0780f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:43.396154 1668 raft_consensus.cc:1261] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Refusing update from remote peer e0da7413c74f48cc9ba5d0e50a0780f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:43.396523 2067 consensus_queue.cc:1035] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:43.396878 2063 consensus_queue.cc:1035] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:43.403939 2023 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: TxnStatusTablet state changed. Reason: New leader e0da7413c74f48cc9ba5d0e50a0780f5. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.404271 2023 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.406163 2072 mvcc.cc:205] Tried to move back new op lower bound from 7334183908942884864 to 7334183908837335040. Current Snapshot: MvccSnapshot[applied={T|T < 7334183908942884864}]
I20260928 05:10:43.407131 2067 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.407456 2067 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:10:43.414517 2023 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.414813 2023 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P 28845aca0f28479dbd5624bfee6c5836: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.419854 2060 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: TxnStatusTablet state changed. Reason: New leader e0da7413c74f48cc9ba5d0e50a0780f5. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.420150 2060 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.421428 2067 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.421734 2067 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P e0da7413c74f48cc9ba5d0e50a0780f5: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:10:43.425217 2074 mvcc.cc:205] Tried to move back new op lower bound from 7334183908942884864 to 7334183908837335040. Current Snapshot: MvccSnapshot[applied={T|T < 7334183908942884864 or (T in {7334183908942884864})}]
I20260928 05:10:43.425272 2060 tablet_replica.cc:439] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 } } peers { permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 } } peers { permanent_uuid: "28845aca0f28479dbd5624bfee6c5836" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42945 } } }
I20260928 05:10:43.425503 2060 tablet_replica.cc:442] T e2c4aa9d0a3341a39efa545ae4978281 P a2f67074a0e443ec8d39f9d2573dfc4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:43.434444 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.421233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.0.97.3:42945)
0928 05:10:43.421416 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.421422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.428220 (+ 6798us) server_negotiation.cc:407] Connection header received
0928 05:10:43.428384 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.428388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.428465 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.428638 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:43.429092 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.429909 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.432287 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.432727 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.432830 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.433291 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.433873 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.433978 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.434113 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.434174 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:10:43.449613 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.443346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51383 (local address 127.0.97.2:42793)
0928 05:10:43.443523 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.443532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.444037 (+ 505us) server_negotiation.cc:407] Connection header received
0928 05:10:43.444238 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.444242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.444309 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.444382 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:43.444915 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.445709 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.448017 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.448480 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.448578 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.449051 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.449143 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.449203 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.449357 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.449414 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:10:43.449903 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.443333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42793 (local address 127.0.97.3:51383)
0928 05:10:43.443938 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:43.443957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:43.444105 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:43.444424 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:43.444429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:43.444443 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:43.444692 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.444703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.445849 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.445854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.447860 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.447875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.448626 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.448634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.448805 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.448873 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.449233 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.449244 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:10:43.449335 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 05:10:43.449475 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
I20260928 05:10:43.458664 1812 raft_consensus.cc:1261] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Refusing update from remote peer 28845aca0f28479dbd5624bfee6c5836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:43.458839 1668 raft_consensus.cc:1261] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Refusing update from remote peer 28845aca0f28479dbd5624bfee6c5836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:43.459264 2023 consensus_queue.cc:1035] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0da7413c74f48cc9ba5d0e50a0780f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:43.459594 2085 consensus_queue.cc:1035] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f67074a0e443ec8d39f9d2573dfc4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:43.496412 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.489703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39163 (local address 127.0.97.3:42945)
0928 05:10:43.489914 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.489923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.490644 (+ 721us) server_negotiation.cc:407] Connection header received
0928 05:10:43.490887 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.490892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.490986 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.491088 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:43.491619 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.492317 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.494462 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.495169 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.495294 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.495828 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.495930 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.495995 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.496140 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.496197 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 05:10:43.497175 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.489611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42945 (local address 127.0.97.2:39163)
0928 05:10:43.490554 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:43.490575 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:43.490759 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:43.491110 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:43.491116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:43.491128 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:43.491481 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.491492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.492440 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.492445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.494314 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:43.494330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.495369 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.495375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:43.495576 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.495650 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:43.496024 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:43.496040 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 05:10:43.496132 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 05:10:43.496288 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":595,"threads_started":1}
I20260928 05:10:43.537330 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:43.537595 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:43.540974 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 840ea87e945a4d67acadcbfac3393652: Sending DeleteTablet for 3 replicas of tablet c9fd45388e3b4bb5aa226bda31d2ee56
I20260928 05:10:43.541844 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fd45388e3b4bb5aa226bda31d2ee56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41642
I20260928 05:10:43.542096 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fd45388e3b4bb5aa226bda31d2ee56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39936
I20260928 05:10:43.542709 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fd45388e3b4bb5aa226bda31d2ee56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57132
I20260928 05:10:43.568269 2097 tablet_replica.cc:330] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: stopping tablet replica
I20260928 05:10:43.568833 2095 tablet_replica.cc:330] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: stopping tablet replica
I20260928 05:10:43.569103 2097 raft_consensus.cc:2229] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:43.569494 2097 raft_consensus.cc:2258] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:43.569739 2095 raft_consensus.cc:2229] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:43.570011 2095 raft_consensus.cc:2258] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:43.571244 2097 ts_tablet_manager.cc:1905] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:43.571751 2096 tablet_replica.cc:330] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: stopping tablet replica
I20260928 05:10:43.571970 2096 raft_consensus.cc:2229] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:10:43.572587 2095 ts_tablet_manager.cc:1905] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:43.572583 2096 raft_consensus.cc:2258] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:43.573712 2096 ts_tablet_manager.cc:1905] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:43.582067 2096 ts_tablet_manager.cc:1918] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:43.582226 2096 log.cc:1196] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/c9fd45388e3b4bb5aa226bda31d2ee56
I20260928 05:10:43.582805 2096 ts_tablet_manager.cc:1939] T c9fd45388e3b4bb5aa226bda31d2ee56 P 28845aca0f28479dbd5624bfee6c5836: Deleting consensus metadata
I20260928 05:10:43.583945 1496 catalog_manager.cc:4841] TS 28845aca0f28479dbd5624bfee6c5836 (127.0.97.3:42945): tablet c9fd45388e3b4bb5aa226bda31d2ee56 (table test-table [id=601a7aee72434eeba0e38a1424cbc0b6]) successfully deleted
I20260928 05:10:43.586153 2095 ts_tablet_manager.cc:1918] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:43.586309 2095 log.cc:1196] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/c9fd45388e3b4bb5aa226bda31d2ee56
I20260928 05:10:43.586784 2095 ts_tablet_manager.cc:1939] T c9fd45388e3b4bb5aa226bda31d2ee56 P a2f67074a0e443ec8d39f9d2573dfc4d: Deleting consensus metadata
I20260928 05:10:43.587860 1495 catalog_manager.cc:4841] TS a2f67074a0e443ec8d39f9d2573dfc4d (127.0.97.1:46859): tablet c9fd45388e3b4bb5aa226bda31d2ee56 (table test-table [id=601a7aee72434eeba0e38a1424cbc0b6]) successfully deleted
I20260928 05:10:43.588629 2097 ts_tablet_manager.cc:1918] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:43.588773 2097 log.cc:1196] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/c9fd45388e3b4bb5aa226bda31d2ee56
I20260928 05:10:43.589272 2097 ts_tablet_manager.cc:1939] T c9fd45388e3b4bb5aa226bda31d2ee56 P e0da7413c74f48cc9ba5d0e50a0780f5: Deleting consensus metadata
I20260928 05:10:43.590386 1495 catalog_manager.cc:4841] TS e0da7413c74f48cc9ba5d0e50a0780f5 (127.0.97.2:42793): tablet c9fd45388e3b4bb5aa226bda31d2ee56 (table test-table [id=601a7aee72434eeba0e38a1424cbc0b6]) successfully deleted
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:43.606323 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.566299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.0.97.1:46859)
0928 05:10:43.566497 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.566503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.575121 (+ 8618us) server_negotiation.cc:407] Connection header received
0928 05:10:43.575403 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.575409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.575491 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.575563 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:43.590298 (+ 14735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.591408 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.593466 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.594101 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.594245 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.599380 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:43.599396 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:43.599417 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:43.599460 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:43.603255 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.604136 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.604145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.604149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.604317 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.604724 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.604729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.604733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.605158 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:43.605474 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.605737 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.605958 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 05:10:43.607055 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41672
I20260928 05:10:43.632176 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.611102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.0.97.62:39459)
0928 05:10:43.611592 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.611599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.614994 (+ 3395us) server_negotiation.cc:407] Connection header received
0928 05:10:43.615158 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.615164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.615253 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.615366 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:43.617084 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.618401 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.620716 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.621451 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.621549 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.624627 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:43.624633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:43.624639 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:43.624678 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:43.628478 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.629331 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.629337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.629341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.629488 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.629919 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.629925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.629929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.630270 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:43.630994 (+ 724us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.631545 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.631769 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":226,"threads_started":1}
W20260928 05:10:43.633353 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40694
I20260928 05:10:43.635547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1582
I20260928 05:10:43.654316 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.637597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.97.1:46859)
0928 05:10:43.638309 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.638315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.638335 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:10:43.638389 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.638394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.638481 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.638657 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:43.640085 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.641188 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.643104 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.643730 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.643827 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.647746 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:43.647752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:43.647757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:43.647794 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:43.651115 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.652103 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.652110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.652113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.652257 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.652675 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.652680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.652684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.652988 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:43.653213 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.653745 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.653961 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
W20260928 05:10:43.655644 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41676
W20260928 05:10:43.656250 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:10:43.656448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1582
I20260928 05:10:43.687642 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1722
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:10:43.707513 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.688436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.0.97.2:42793)
0928 05:10:43.688631 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.688637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.688707 (+ 70us) server_negotiation.cc:407] Connection header received
0928 05:10:43.688925 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.688932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.689011 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.689136 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:43.690616 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.691810 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.693654 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.694306 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.694443 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.699757 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:43.699772 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:43.699795 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:43.699861 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:43.704167 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.705015 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.705025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.705029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.705223 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.705627 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.705633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.705636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.706068 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:43.706400 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.706716 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.706941 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 05:10:43.710058 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57146
W20260928 05:10:43.710906 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:10:43.711056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1722
I20260928 05:10:43.730901 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1861
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:10:43.747802 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:43.731679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.97.3:42945)
0928 05:10:43.731886 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:43.731892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:43.731912 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:10:43.732064 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:43.732070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:43.732164 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:43.732284 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:43.733428 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.734649 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.735841 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:43.736558 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:43.736696 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:43.740903 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:43.740919 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:43.740945 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:43.740989 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:43.744875 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.745542 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.745551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.745555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.745727 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:43.746055 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:43.746060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:43.746064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:43.746588 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:43.746864 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:43.747199 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 05:10:43.747461 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 05:10:43.748546 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39956
W20260928 05:10:43.749240 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:10:43.749413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1861
I20260928 05:10:43.767701 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 1479
I20260928 05:10:43.768795 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:43.880501 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 1479
2026-09-28T05:10:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4913 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[2107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[2107](info): set up 2 sockets
Sep 28 05:10:43 dist-test-slave-6718 krb5kdc[2107](info): commencing operation
krb5kdc: starting...
W20260928 05:10:44.958782 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:10:45 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:10:45.135281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45517
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:45.462363 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:45.462774 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:45.462937 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:45.470186 2120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:45.470336 2120 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:10:45.470440 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:45.470572 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:45.470646 2120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:45.470739 2120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:45.480543 2120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:45.481840 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:45.483232 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:45.494589 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:45.494547 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:45.494626 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:45.495464 2120 server_base.cc:1034] running on GCE node
I20260928 05:10:45.496260 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:45.497557 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:45.498801 2120 hybrid_clock.cc:648] HybridClock initialized: now 1790572245498725 us; error 54 us; skew 500 ppm
I20260928 05:10:45.499120 2120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:45 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:45.502027 2120 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:45.503441 2120 webserver.cc:460] Webserver started at http://127.0.97.62:35633/ using document root <none> and password file <none>
I20260928 05:10:45.504119 2120 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:45.504252 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:45.504555 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:45.507404 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "1345207eaa2b46e38ae72768434b57a5"
format_stamp: "Formatted at 2026-09-28 05:10:45 on dist-test-slave-6718"
server_key: "e06daf691befff93b606ea86e0f6f154"
server_key_iv: "41733f1426686494c60b9d3443daca37"
server_key_version: "encryptionkey@0"
I20260928 05:10:45.508152 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "1345207eaa2b46e38ae72768434b57a5"
format_stamp: "Formatted at 2026-09-28 05:10:45 on dist-test-slave-6718"
server_key: "e06daf691befff93b606ea86e0f6f154"
server_key_iv: "41733f1426686494c60b9d3443daca37"
server_key_version: "encryptionkey@0"
I20260928 05:10:45.513096 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:10:45.516878 2135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:45.518146 2120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:10:45.518332 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "1345207eaa2b46e38ae72768434b57a5"
format_stamp: "Formatted at 2026-09-28 05:10:45 on dist-test-slave-6718"
server_key: "e06daf691befff93b606ea86e0f6f154"
server_key_iv: "41733f1426686494c60b9d3443daca37"
server_key_version: "encryptionkey@0"
I20260928 05:10:45.518539 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:45.546483 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:45.550239 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:45.550516 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:45.571799 2120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45517
I20260928 05:10:45.571817 2196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45517 every 8 connection(s)
I20260928 05:10:45.573117 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:10:45.576844 2197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:45.580708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 2120
I20260928 05:10:45.580925 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:10:45.581674 388 external_mini_cluster.cc:1428] Setting key ca47854331c5d5b99c2cc0accadcdb7e
I20260928 05:10:45.589253 2197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Bootstrap starting.
I20260928 05:10:45.592307 2197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:45.593258 2197 log.cc:824] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:10:45 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:45.595824 2197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: No bootstrap required, opened a new log
I20260928 05:10:45.602123 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:45.583753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.97.62:45517)
0928 05:10:45.584433 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:45.584443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:45.584481 (+ 38us) server_negotiation.cc:407] Connection header received
0928 05:10:45.585937 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:45.585960 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:45.586372 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:45.586810 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:45.588154 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:45.589458 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:45.590670 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:45.591327 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:45.591509 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:45.595376 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:45.595383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:45.595396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:45.595428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:45.598327 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:45.599017 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:45.599031 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:45.599037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:45.599210 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:45.599579 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:45.599588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:45.599590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:45.600079 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:45.600389 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:45.600808 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 05:10:45.601196 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":296,"threads_started":1}
I20260928 05:10:45.603618 2197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1345207eaa2b46e38ae72768434b57a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260928 05:10:45.603955 2197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:45.604072 2197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1345207eaa2b46e38ae72768434b57a5, State: Initialized, Role: FOLLOWER
I20260928 05:10:45.604604 2197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1345207eaa2b46e38ae72768434b57a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260928 05:10:45.604885 2197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:10:45.605020 2197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:10:45.605154 2197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:45.607213 2197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1345207eaa2b46e38ae72768434b57a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260928 05:10:45.607627 2197 leader_election.cc:302] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1345207eaa2b46e38ae72768434b57a5; no voters:
I20260928 05:10:45.608139 2197 leader_election.cc:288] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:10:45.608273 2202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:45.609037 2202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1345207eaa2b46e38ae72768434b57a5, State: Running, Role: LEADER
I20260928 05:10:45.609570 2202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1345207eaa2b46e38ae72768434b57a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } }
I20260928 05:10:45.610457 2197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:10:45.612876 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1345207eaa2b46e38ae72768434b57a5. Latest consensus state: current_term: 1 leader_uuid: "1345207eaa2b46e38ae72768434b57a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1345207eaa2b46e38ae72768434b57a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } } }
I20260928 05:10:45.613065 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1345207eaa2b46e38ae72768434b57a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1345207eaa2b46e38ae72768434b57a5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45517 } } }
I20260928 05:10:45.614322 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:45.614357 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:45.618355 2210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:10:45.623097 2210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:10:45.630882 2210 catalog_manager.cc:1347] Generated new cluster ID: 719f176c8c5744dfad729b4d155e4eda
I20260928 05:10:45.630976 2210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:10:45.640789 2210 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:10:45.642483 2210 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:10:45.652666 2210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Generated new TSK 0
I20260928 05:10:45.653613 2210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:10:45.699028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45517
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:46.019970 2223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:46.020244 2223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:46.020409 2223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:46.027948 2223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:46.028066 2223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:46.028209 2223 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:10:46.028286 2223 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:10:46.028422 2223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:10:46.038065 2223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:46.039376 2223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:46.040640 2223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:46.052839 2228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:46.052884 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:46.052901 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:46.053933 2223 server_base.cc:1034] running on GCE node
I20260928 05:10:46.054507 2223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:46.055441 2223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:46.056672 2223 hybrid_clock.cc:648] HybridClock initialized: now 1790572246056619 us; error 44 us; skew 500 ppm
I20260928 05:10:46.056990 2223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:46 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:46.060067 2223 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:10:46.061450 2223 webserver.cc:460] Webserver started at http://127.0.97.1:35943/ using document root <none> and password file <none>
I20260928 05:10:46.062152 2223 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:46.062305 2223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:46.062681 2223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:46.065531 2223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "d8c30be52ff3428385316a0b16af9d1f"
format_stamp: "Formatted at 2026-09-28 05:10:46 on dist-test-slave-6718"
server_key: "337374c66a7f2565ac56826cb776e9c5"
server_key_iv: "a05f39ab12fe988ba0ca5ac812c3939a"
server_key_version: "encryptionkey@0"
I20260928 05:10:46.066308 2223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d8c30be52ff3428385316a0b16af9d1f"
format_stamp: "Formatted at 2026-09-28 05:10:46 on dist-test-slave-6718"
server_key: "337374c66a7f2565ac56826cb776e9c5"
server_key_iv: "a05f39ab12fe988ba0ca5ac812c3939a"
server_key_version: "encryptionkey@0"
I20260928 05:10:46.071434 2223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 05:10:46.074862 2238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:46.076189 2223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:10:46.076354 2223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "d8c30be52ff3428385316a0b16af9d1f"
format_stamp: "Formatted at 2026-09-28 05:10:46 on dist-test-slave-6718"
server_key: "337374c66a7f2565ac56826cb776e9c5"
server_key_iv: "a05f39ab12fe988ba0ca5ac812c3939a"
server_key_version: "encryptionkey@0"
I20260928 05:10:46.076543 2223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:46.110257 2223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:46.115242 2223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:46.115535 2223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:46.118125 2223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:46.118235 2223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:46.118372 2223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:46.118464 2223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:10:46 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:46.143244 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.119057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59791 (local address 127.0.97.62:45517)
0928 05:10:46.119506 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:46.119512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:46.120746 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 05:10:46.123475 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:46.123484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:46.123600 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:46.123735 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:46.125933 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.126963 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.129480 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.129870 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.129999 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.134678 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:46.134683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:46.134687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:46.134717 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:46.138667 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.139484 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.139492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.139498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.139652 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.140223 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.140231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.140236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.140625 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:46.140766 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:46.142709 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0928 05:10:46.142928 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260928 05:10:46.143746 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.119704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.97.1:59791)
0928 05:10:46.120558 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:46.120613 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:46.123169 (+ 2556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:46.123985 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:46.123993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:46.124761 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:46.125726 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.125759 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.127126 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.127130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.129277 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.129314 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.130121 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.130127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.130472 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.131466 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:46.131490 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:46.134438 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:46.138859 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.138869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.138889 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.139363 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.139787 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.139792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.139798 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.140087 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.140738 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:46.140753 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:46.141250 (+ 497us) client_negotiation.cc:769] Sending connection context
0928 05:10:46.141566 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 05:10:46.142084 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260928 05:10:46.160730 2223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43827
I20260928 05:10:46.160765 2358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43827 every 8 connection(s)
I20260928 05:10:46.161927 2223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:10:46.164068 2359 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45517
I20260928 05:10:46.164314 2359 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:46.164979 2359 heartbeater.cc:511] Master 127.0.97.62:45517 requested a full tablet report, sending...
I20260928 05:10:46.165155 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.150978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55667 (local address 127.0.97.62:45517)
0928 05:10:46.151134 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:46.151138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:46.151902 (+ 764us) server_negotiation.cc:407] Connection header received
0928 05:10:46.152175 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:46.152180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:46.152252 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:46.152372 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:46.153861 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.154966 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.157148 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.157707 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.157851 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.159596 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:46.159602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:46.159608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:46.159646 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:46.162262 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.163295 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.163300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.163304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.163434 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.163787 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.163793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.163796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.164045 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:46.164200 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:46.164617 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 05:10:46.164819 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:10:46.165318 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.151057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.97.1:55667)
0928 05:10:46.151811 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:46.151835 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:46.152037 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:46.152557 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:46.152579 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:46.153195 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:46.153725 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.153735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.155175 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.155180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.156998 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.157009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.157892 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.157898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.158053 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.158693 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:46.158712 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:46.159424 (+ 712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:46.162445 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.162453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.162458 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.163148 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.163551 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.163556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.163560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.163680 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.164170 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:46.164177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:46.164390 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 05:10:46.164534 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:10:46.164852 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260928 05:10:46.167183 2151 ts_manager.cc:194] Registered new tserver with Master: d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827)
I20260928 05:10:46.167971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 2223
I20260928 05:10:46.168190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:10:46.168926 388 external_mini_cluster.cc:1428] Setting key 19595eec40550f4f867ca8469d5cc3ef
I20260928 05:10:46.169202 2151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59791
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:10:46.209482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45517
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:46.524080 2363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:46.524339 2363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:46.524531 2363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:46.531585 2363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:46.531695 2363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:46.531841 2363 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:10:46.531921 2363 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:10:46.532033 2363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:10:46.541481 2363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:46.542757 2363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:46.544031 2363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:46.556749 2368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:46.556623 2369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:46.556654 2371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:46.557533 2363 server_base.cc:1034] running on GCE node
I20260928 05:10:46.558106 2363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:46.559017 2363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:46.560201 2363 hybrid_clock.cc:648] HybridClock initialized: now 1790572246560141 us; error 61 us; skew 500 ppm
I20260928 05:10:46.560495 2363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:46 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:46.563472 2363 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:10:46.564875 2363 webserver.cc:460] Webserver started at http://127.0.97.2:36471/ using document root <none> and password file <none>
I20260928 05:10:46.565467 2363 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:46.565563 2363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:46.565795 2363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:46.568578 2363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "780719235a2447119eff68a541837593"
format_stamp: "Formatted at 2026-09-28 05:10:46 on dist-test-slave-6718"
server_key: "87c070a880e85cc7daee59e982aac46a"
server_key_iv: "5fe7f8f5f663a2f93198adb31e09166d"
server_key_version: "encryptionkey@0"
I20260928 05:10:46.569279 2363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "780719235a2447119eff68a541837593"
format_stamp: "Formatted at 2026-09-28 05:10:46 on dist-test-slave-6718"
server_key: "87c070a880e85cc7daee59e982aac46a"
server_key_iv: "5fe7f8f5f663a2f93198adb31e09166d"
server_key_version: "encryptionkey@0"
I20260928 05:10:46.575268 2363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 05:10:46.579350 2378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:46.580952 2363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:10:46.581094 2363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "780719235a2447119eff68a541837593"
format_stamp: "Formatted at 2026-09-28 05:10:46 on dist-test-slave-6718"
server_key: "87c070a880e85cc7daee59e982aac46a"
server_key_iv: "5fe7f8f5f663a2f93198adb31e09166d"
server_key_version: "encryptionkey@0"
I20260928 05:10:46.581285 2363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:46.605377 2363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:46.612891 2363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:46.613130 2363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:46.615885 2363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:46.616001 2363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:46.616117 2363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:46.616189 2363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:10:46 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:46.640254 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.616492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41993 (local address 127.0.97.62:45517)
0928 05:10:46.616653 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:46.616658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:46.618117 (+ 1459us) server_negotiation.cc:407] Connection header received
0928 05:10:46.620460 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:46.620465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:46.620555 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:46.620668 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:46.623142 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.624205 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.626917 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.627544 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.627679 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.632825 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:46.632833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:46.632841 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:46.632900 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:46.636726 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.637446 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.637454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.637460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.637626 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.638198 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.638204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.638207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.638627 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:46.638785 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:46.639685 (+ 900us) server_negotiation.cc:299] Negotiation successful
0928 05:10:46.639916 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:10:46.641233 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.617088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.97.2:41993)
0928 05:10:46.617952 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:46.617990 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:46.620183 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:46.620932 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:46.620946 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:46.621947 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:46.622938 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.622967 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.624371 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.624377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.626751 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.626763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.627795 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.627804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.628150 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.628991 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:46.629034 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:46.632564 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:46.636895 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.636902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.636914 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.637313 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.637757 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.637764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.637770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.638060 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.638758 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:46.638769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:46.639260 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 05:10:46.639604 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 05:10:46.640102 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":365,"threads_started":1}
I20260928 05:10:46.663141 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.648201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37577 (local address 127.0.97.62:45517)
0928 05:10:46.648433 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:46.648441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:46.649167 (+ 726us) server_negotiation.cc:407] Connection header received
0928 05:10:46.649333 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:46.649340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:46.649438 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:46.649548 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:46.650440 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.651725 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.653627 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.654203 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.654339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.657041 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:46.657047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:46.657052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:46.657091 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:46.660517 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.661196 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.661211 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.661215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.661332 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:46.661683 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:46.661690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:46.661693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:46.661992 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:46.662176 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:46.662586 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 05:10:46.662799 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:10:46.663398 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:46.648427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.97.2:37577)
0928 05:10:46.649061 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:46.649079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:46.649205 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:46.649585 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:46.649588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:46.650027 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:46.650299 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.650308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.651897 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.651903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.653362 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:46.653374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:46.654599 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:46.654608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:46.654860 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:46.655585 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:46.655614 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:46.656792 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:46.660643 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.660651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.660655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.661079 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.661450 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:46.661454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:46.661458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:46.661581 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:46.662144 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:46.662148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:46.662322 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 05:10:46.662466 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:10:46.662886 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":339,"threads_started":1}
I20260928 05:10:46.669620 2363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40269
I20260928 05:10:46.669655 2497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40269 every 8 connection(s)
I20260928 05:10:46.670881 2363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:10:46.672796 2498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45517
I20260928 05:10:46.673005 2498 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:46.673620 2498 heartbeater.cc:511] Master 127.0.97.62:45517 requested a full tablet report, sending...
I20260928 05:10:46.674924 2152 ts_manager.cc:194] Registered new tserver with Master: 780719235a2447119eff68a541837593 (127.0.97.2:40269)
I20260928 05:10:46.675741 2152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41993
I20260928 05:10:46.676496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 2363
I20260928 05:10:46.676680 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:10:46.677089 388 external_mini_cluster.cc:1428] Setting key adea5a82aac276edf0c473c3a880ee40
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:10:46.717607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45517
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:47.029326 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:47.029600 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:47.029700 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:47.036945 2502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:47.037070 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:47.037156 2502 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 05:10:47.037209 2502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:10:47.037271 2502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:10:47.047433 2502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:47.048658 2502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:47.050020 2502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:47.062831 2507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:47.062898 2508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:47.062875 2510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:47.063673 2502 server_base.cc:1034] running on GCE node
I20260928 05:10:47.064313 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:47.065196 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:47.066392 2502 hybrid_clock.cc:648] HybridClock initialized: now 1790572247066348 us; error 45 us; skew 500 ppm
I20260928 05:10:47.066725 2502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:47 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:47.069646 2502 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:10:47.071152 2502 webserver.cc:460] Webserver started at http://127.0.97.3:39251/ using document root <none> and password file <none>
I20260928 05:10:47.071810 2502 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:47.071947 2502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:47.072254 2502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:47.075042 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "45a665d368f54cf79494f6ead7d38f31"
format_stamp: "Formatted at 2026-09-28 05:10:47 on dist-test-slave-6718"
server_key: "2a2e9f0ba7f9492a9c94fcdcfe38011c"
server_key_iv: "1e9914560c76373db0a551e16537982d"
server_key_version: "encryptionkey@0"
I20260928 05:10:47.075794 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "45a665d368f54cf79494f6ead7d38f31"
format_stamp: "Formatted at 2026-09-28 05:10:47 on dist-test-slave-6718"
server_key: "2a2e9f0ba7f9492a9c94fcdcfe38011c"
server_key_iv: "1e9914560c76373db0a551e16537982d"
server_key_version: "encryptionkey@0"
I20260928 05:10:47.080701 2502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:10:47.084057 2517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:47.085336 2502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:10:47.085505 2502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "45a665d368f54cf79494f6ead7d38f31"
format_stamp: "Formatted at 2026-09-28 05:10:47 on dist-test-slave-6718"
server_key: "2a2e9f0ba7f9492a9c94fcdcfe38011c"
server_key_iv: "1e9914560c76373db0a551e16537982d"
server_key_version: "encryptionkey@0"
I20260928 05:10:47.085758 2502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:47.099524 2502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:47.103247 2502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:47.103525 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:47.105989 2502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:47.106134 2502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:47.106258 2502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:47.106359 2502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:10:47 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:47.130122 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.106811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37591 (local address 127.0.97.62:45517)
0928 05:10:47.106994 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.107000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.108354 (+ 1354us) server_negotiation.cc:407] Connection header received
0928 05:10:47.110778 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.110782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.110895 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.110954 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:47.113487 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.114621 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.117082 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.117671 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.117853 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.122826 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:47.122836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:47.122843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:47.122888 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:47.126117 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.127013 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.127020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.127026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.127185 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.127874 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.127881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.127883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.128175 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:47.128359 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.129421 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.129682 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:10:47.131965 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.107292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.97.3:37591)
0928 05:10:47.108165 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.108218 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.110386 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.111504 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.111520 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.112298 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:47.113276 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.113301 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.114790 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.114793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.116912 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.116927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.118151 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.118156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.118435 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.119259 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:47.119308 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:47.122467 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:47.126328 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:47.126338 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:47.126362 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:47.126893 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:47.127329 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:47.127338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:47.127344 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:47.127665 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:47.128333 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:47.128345 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:47.128867 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.129272 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.129946 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":378,"threads_started":1}
I20260928 05:10:47.151547 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.137363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.97.3:45145)
0928 05:10:47.137966 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.137983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.138127 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.138430 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.138434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.139023 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:47.139416 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.139428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.140367 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.140373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.141349 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.141359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.142215 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.142221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.142372 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.143002 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:47.143030 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:47.144435 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:47.148568 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:47.148575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:47.148580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:47.149159 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:47.149600 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:47.149607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:47.149614 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:47.149795 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:47.150490 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:47.150494 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:47.150695 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.150870 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.151214 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":361,"threads_started":1}
I20260928 05:10:47.152408 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.137272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45145 (local address 127.0.97.62:45517)
0928 05:10:47.137418 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.137422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.138062 (+ 640us) server_negotiation.cc:407] Connection header received
0928 05:10:47.138243 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.138252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.138314 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.138410 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:47.139555 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.140237 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.141482 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.142036 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.142179 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.144633 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:47.144643 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:47.144649 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:47.144695 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:47.148427 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.149291 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.149299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.149305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.149481 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.149966 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.149976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.149978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.150261 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:47.150419 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.151854 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.152087 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 05:10:47.155437 2502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37721
I20260928 05:10:47.155520 2636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37721 every 8 connection(s)
I20260928 05:10:47.156816 2502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:10:47.158634 2637 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45517
I20260928 05:10:47.158872 2637 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:47.159467 2637 heartbeater.cc:511] Master 127.0.97.62:45517 requested a full tablet report, sending...
I20260928 05:10:47.160923 2152 ts_manager.cc:194] Registered new tserver with Master: 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:47.162048 2152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37591
I20260928 05:10:47.164750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 2502
I20260928 05:10:47.164953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:10:47.165359 388 external_mini_cluster.cc:1428] Setting key 0004b5218dd36300b6bed6f6d4122b36
I20260928 05:10:47.168177 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:10:47.171952 2359 heartbeater.cc:503] Master 127.0.97.62:45517 was elected leader, sending a full tablet report...
Sep 28 05:10:47 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:10:47 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:47.214393 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.201350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.97.62:45517)
0928 05:10:47.201525 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.201529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.201859 (+ 330us) server_negotiation.cc:407] Connection header received
0928 05:10:47.202000 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.202004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.202065 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.202157 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:47.203535 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.204267 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.205431 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.205841 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.205939 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.209573 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:47.209578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:47.209581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:47.209619 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:47.212328 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.212905 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.212909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.212911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.213021 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.213316 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.213319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.213321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.213547 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:47.213725 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.213974 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.214132 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:10:47.218293 2152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:10:47.225317 2152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:47.260581 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.237085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37721 (local address 127.0.0.1:46246)
0928 05:10:47.238656 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.238702 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.238860 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.251235 (+ 12375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.251247 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.251292 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:47.251816 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.251832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.254854 (+ 3022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.254863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.257445 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.257459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.259583 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.259590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.259820 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.259871 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.259969 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.260144 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"thread_start_us":1151,"threads_started":1}
I20260928 05:10:47.261994 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.237386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.0.97.3:37721)
0928 05:10:47.250111 (+ 12725us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.250124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.250165 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:10:47.250296 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.250308 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.250647 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.252114 (+ 1467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:47.252466 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.254641 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.257645 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.259357 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.261212 (+ 1855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.261447 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.261518 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.261667 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12536,"thread_start_us":12373,"threads_started":1}
I20260928 05:10:47.264940 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.237301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40269 (local address 127.0.0.1:38154)
0928 05:10:47.248901 (+ 11600us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.248925 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.249067 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.257877 (+ 8810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.257882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.257910 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:47.258326 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.258337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.260338 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.260343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.262450 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.262465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.264223 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.264230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.264387 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.264427 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.264494 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.264620 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11425,"thread_start_us":13560,"threads_started":1}
I20260928 05:10:47.266197 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.237914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.97.2:40269)
0928 05:10:47.256182 (+ 18268us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.256191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.256232 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:10:47.256333 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.256341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.257637 (+ 1296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.258520 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:47.258742 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.260175 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.262666 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.264008 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.265452 (+ 1444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.265652 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.265731 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.265906 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18099,"thread_start_us":19389,"threads_started":1}
I20260928 05:10:47.267870 2567 tablet_service.cc:1467] Processing CreateTablet for tablet 764e3c36cc47420da635413eac70748e (DEFAULT_TABLE table=test-table [id=25d20a73db2b4b54b01f04826cac5320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:47.269430 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764e3c36cc47420da635413eac70748e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:47.269739 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 764e3c36cc47420da635413eac70748e (DEFAULT_TABLE table=test-table [id=25d20a73db2b4b54b01f04826cac5320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:47.271047 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764e3c36cc47420da635413eac70748e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:47.272312 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.239060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.0.1:52418)
0928 05:10:47.260935 (+ 21875us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.260960 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.261048 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.263372 (+ 2324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.263380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.263410 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:47.263609 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.263622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.265643 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.265652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.269179 (+ 3527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.269198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.271362 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.271371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.271612 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.271667 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.271741 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.271928 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21780,"thread_start_us":24469,"threads_started":1}
I20260928 05:10:47.273463 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.239358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.0.97.1:43827)
0928 05:10:47.262813 (+ 23455us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.262821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.262847 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:10:47.262933 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.262941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.263168 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.263357 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:47.263791 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.265467 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.269420 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.271180 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.272571 (+ 1391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.272839 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.272911 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.273104 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23251,"thread_start_us":23105,"threads_started":1}
I20260928 05:10:47.277382 2289 tablet_service.cc:1467] Processing CreateTablet for tablet 764e3c36cc47420da635413eac70748e (DEFAULT_TABLE table=test-table [id=25d20a73db2b4b54b01f04826cac5320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:47.278855 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764e3c36cc47420da635413eac70748e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:47.285971 2656 tablet_bootstrap.cc:492] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Bootstrap starting.
I20260928 05:10:47.288090 2657 tablet_bootstrap.cc:492] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Bootstrap starting.
I20260928 05:10:47.290369 2656 tablet_bootstrap.cc:654] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:47.291355 2657 tablet_bootstrap.cc:654] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:47.291791 2656 log.cc:824] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:47.292634 2657 log.cc:824] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:47.294624 2660 tablet_bootstrap.cc:492] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Bootstrap starting.
I20260928 05:10:47.295907 2656 tablet_bootstrap.cc:492] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: No bootstrap required, opened a new log
I20260928 05:10:47.296036 2657 tablet_bootstrap.cc:492] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: No bootstrap required, opened a new log
I20260928 05:10:47.296234 2656 ts_tablet_manager.cc:1397] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 05:10:47.296356 2657 ts_tablet_manager.cc:1397] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 05:10:47.298410 2660 tablet_bootstrap.cc:654] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:47.299768 2660 log.cc:824] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:47.303025 2660 tablet_bootstrap.cc:492] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: No bootstrap required, opened a new log
I20260928 05:10:47.303316 2660 ts_tablet_manager.cc:1397] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 05:10:47.305527 2657 raft_consensus.cc:348] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.305943 2657 raft_consensus.cc:374] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:47.306051 2657 raft_consensus.cc:729] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780719235a2447119eff68a541837593, State: Initialized, Role: FOLLOWER
I20260928 05:10:47.306551 2656 raft_consensus.cc:348] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.306759 2657 consensus_queue.cc:260] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.306957 2656 raft_consensus.cc:374] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:47.307099 2656 raft_consensus.cc:729] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a665d368f54cf79494f6ead7d38f31, State: Initialized, Role: FOLLOWER
I20260928 05:10:47.307698 2656 consensus_queue.cc:260] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.308804 2637 heartbeater.cc:503] Master 127.0.97.62:45517 was elected leader, sending a full tablet report...
I20260928 05:10:47.310909 2656 ts_tablet_manager.cc:1428] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260928 05:10:47.313256 2660 raft_consensus.cc:348] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.313613 2660 raft_consensus.cc:374] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:47.313736 2660 raft_consensus.cc:729] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Initialized, Role: FOLLOWER
I20260928 05:10:47.314291 2660 consensus_queue.cc:260] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.315310 2498 heartbeater.cc:503] Master 127.0.97.62:45517 was elected leader, sending a full tablet report...
I20260928 05:10:47.317233 2660 ts_tablet_manager.cc:1428] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260928 05:10:47.319298 2657 ts_tablet_manager.cc:1428] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Time spent starting tablet: real 0.023s user 0.010s sys 0.010s
W20260928 05:10:47.408371 2638 tablet.cc:2369] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:10:47.414409 2360 tablet.cc:2369] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:10:47.422755 2499 tablet.cc:2369] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:10:47.480835 2664 raft_consensus.cc:482] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:47.481159 2664 raft_consensus.cc:504] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.482801 2664 leader_election.cc:288] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721), 780719235a2447119eff68a541837593 (127.0.97.2:40269)
I20260928 05:10:47.490692 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.483174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45941 (local address 127.0.97.3:37721)
0928 05:10:47.483565 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.483573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.483902 (+ 329us) server_negotiation.cc:407] Connection header received
0928 05:10:47.484165 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.484170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.484237 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.484302 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:47.484927 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.485702 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.488790 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.489725 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.489894 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.490000 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.490267 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.490397 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260928 05:10:47.490733 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.483012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37721 (local address 127.0.97.1:45941)
0928 05:10:47.483786 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.483805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.483979 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.484360 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.484364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.484386 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:47.484780 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.484791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.485873 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.485881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.488657 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.488669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.489952 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.489956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.490070 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.490119 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.490212 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.490402 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":210,"thread_start_us":445,"threads_started":1}
I20260928 05:10:47.491657 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.483224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57231 (local address 127.0.97.2:40269)
0928 05:10:47.483446 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.483452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.484027 (+ 575us) server_negotiation.cc:407] Connection header received
0928 05:10:47.484194 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.484197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.484253 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.484353 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:47.485033 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.485847 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.489552 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.490700 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.490846 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.490964 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.491362 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.491486 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:10:47.491770 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.483189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40269 (local address 127.0.97.1:57231)
0928 05:10:47.483925 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.483939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.484043 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.484389 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.484393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.484415 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:47.484844 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.484877 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.485946 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.485950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.488654 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.488669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.491086 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.491091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.491203 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.491226 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.491298 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.491383 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":205,"thread_start_us":458,"threads_started":1}
I20260928 05:10:47.491936 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764e3c36cc47420da635413eac70748e" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31" is_pre_election: true
I20260928 05:10:47.492554 2587 raft_consensus.cc:2454] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:47.492863 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764e3c36cc47420da635413eac70748e" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593" is_pre_election: true
I20260928 05:10:47.493311 2241 leader_election.cc:302] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:47.493333 2453 raft_consensus.cc:2454] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:47.493687 2664 raft_consensus.cc:2781] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:47.493851 2664 raft_consensus.cc:482] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:47.493979 2664 raft_consensus.cc:3037] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:47.496183 2664 raft_consensus.cc:504] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.496752 2664 leader_election.cc:288] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Requested vote from peers 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721), 780719235a2447119eff68a541837593 (127.0.97.2:40269)
I20260928 05:10:47.497349 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764e3c36cc47420da635413eac70748e" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593"
I20260928 05:10:47.497432 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764e3c36cc47420da635413eac70748e" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31"
I20260928 05:10:47.497553 2453 raft_consensus.cc:3037] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:47.497658 2587 raft_consensus.cc:3037] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:47.499935 2587 raft_consensus.cc:2454] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:47.500161 2453 raft_consensus.cc:2454] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:47.500340 2241 leader_election.cc:302] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:47.500633 2664 raft_consensus.cc:2781] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:47.501062 2664 raft_consensus.cc:686] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Becoming Leader. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Running, Role: LEADER
I20260928 05:10:47.501583 2664 consensus_queue.cc:237] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } }
I20260928 05:10:47.506191 2151 catalog_manager.cc:5495] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8c30be52ff3428385316a0b16af9d1f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } health_report { overall_health: HEALTHY } } }
Sep 28 05:10:47 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:47.573356 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.560761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.97.62:45517)
0928 05:10:47.560931 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.560935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.561450 (+ 515us) server_negotiation.cc:407] Connection header received
0928 05:10:47.561645 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.561651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.561710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.561829 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:47.562990 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.563666 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.564872 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.565291 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.565428 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.568512 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:47.568517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:47.568520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:47.568546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:47.570776 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.571436 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.571443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.571446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.571592 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:47.571961 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:47.571966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:47.571976 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:47.572204 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:47.572353 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.572742 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.572972 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:10:47.573500 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.560795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45517 (local address 127.0.0.1:57748)
0928 05:10:47.561352 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.561366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.561529 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.561838 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.561842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.562514 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:47.562846 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.562855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.563794 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.563801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.564735 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.564746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.565466 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.565472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.565643 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.566274 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:47.566291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:47.568349 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:47.570899 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:47.570907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:47.570912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:47.571289 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:47.571702 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:47.571710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:47.571713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:47.571853 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:47.572348 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:47.572355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:47.572502 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 05:10:47.572666 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 05:10:47.573079 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":162,"thread_start_us":312,"threads_started":1}
I20260928 05:10:47.578449 2151 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:10:47.579600 2151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:47.586319 2289 tablet_service.cc:1467] Processing CreateTablet for tablet fe104e28012246e897effd6374e9922c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dac14d05770d45bb9870a949c01770d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:10:47.586506 2567 tablet_service.cc:1467] Processing CreateTablet for tablet fe104e28012246e897effd6374e9922c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dac14d05770d45bb9870a949c01770d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:10:47.586416 2428 tablet_service.cc:1467] Processing CreateTablet for tablet fe104e28012246e897effd6374e9922c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dac14d05770d45bb9870a949c01770d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:10:47.587224 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe104e28012246e897effd6374e9922c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:47.587289 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe104e28012246e897effd6374e9922c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:47.587289 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe104e28012246e897effd6374e9922c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:47.592314 2660 tablet_bootstrap.cc:492] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: Bootstrap starting.
I20260928 05:10:47.593837 2660 tablet_bootstrap.cc:654] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:47.594187 2657 tablet_bootstrap.cc:492] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: Bootstrap starting.
I20260928 05:10:47.595757 2657 tablet_bootstrap.cc:654] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:47.596030 2660 tablet_bootstrap.cc:492] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: No bootstrap required, opened a new log
I20260928 05:10:47.596143 2660 ts_tablet_manager.cc:1397] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 05:10:47.596835 2660 raft_consensus.cc:348] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.597100 2660 raft_consensus.cc:374] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:47.597224 2660 raft_consensus.cc:729] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Initialized, Role: FOLLOWER
I20260928 05:10:47.597316 2656 tablet_bootstrap.cc:492] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: Bootstrap starting.
I20260928 05:10:47.597532 2660 consensus_queue.cc:260] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.597987 2664 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:47.598163 2657 tablet_bootstrap.cc:492] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: No bootstrap required, opened a new log
I20260928 05:10:47.598235 2660 ts_tablet_manager.cc:1428] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:10:47.598296 2657 ts_tablet_manager.cc:1397] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 05:10:47.598294 2664 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:47.598977 2656 tablet_bootstrap.cc:654] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:47.598929 2657 raft_consensus.cc:348] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.599136 2657 raft_consensus.cc:374] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:47.599202 2657 raft_consensus.cc:729] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780719235a2447119eff68a541837593, State: Initialized, Role: FOLLOWER
I20260928 05:10:47.599399 2657 consensus_queue.cc:260] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.600028 2657 ts_tablet_manager.cc:1428] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 05:10:47.599799 2662 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:47.600399 2662 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:47.607647 2656 tablet_bootstrap.cc:492] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: No bootstrap required, opened a new log
I20260928 05:10:47.607812 2656 ts_tablet_manager.cc:1397] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260928 05:10:47.608474 2656 raft_consensus.cc:348] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.608678 2656 raft_consensus.cc:374] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:47.608784 2656 raft_consensus.cc:729] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a665d368f54cf79494f6ead7d38f31, State: Initialized, Role: FOLLOWER
I20260928 05:10:47.609000 2656 consensus_queue.cc:260] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.609541 2656 ts_tablet_manager.cc:1428] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:10:47.609422 2663 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:47.609757 2663 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:47.980566 2664 consensus_queue.cc:1035] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:47.988569 2662 raft_consensus.cc:482] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:47.988679 2664 consensus_queue.cc:1035] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:47.988804 2662 raft_consensus.cc:504] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.991420 2662 leader_election.cc:288] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:47.994568 2664 raft_consensus.cc:482] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:47.994716 2664 raft_consensus.cc:504] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:47.995282 2664 leader_election.cc:288] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780719235a2447119eff68a541837593 (127.0.97.2:40269), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:47.995793 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe104e28012246e897effd6374e9922c" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31" is_pre_election: true
I20260928 05:10:47.995716 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe104e28012246e897effd6374e9922c" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593" is_pre_election: true
I20260928 05:10:47.996027 2587 raft_consensus.cc:2454] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:47.996167 2452 raft_consensus.cc:2454] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:47.996474 2241 leader_election.cc:302] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:47.996726 2664 raft_consensus.cc:2781] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:47.996810 2664 raft_consensus.cc:482] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:47.996909 2664 raft_consensus.cc:3037] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:47.999887 2664 raft_consensus.cc:504] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.000118 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.990973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36193 (local address 127.0.97.1:43827)
0928 05:10:47.991418 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.991422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:47.992123 (+ 701us) server_negotiation.cc:407] Connection header received
0928 05:10:47.992386 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:47.992392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:47.992464 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:47.992532 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:47.993177 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.994038 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.997541 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.998895 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.998988 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.999084 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:47.999575 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 05:10:47.999747 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":276,"threads_started":1}
I20260928 05:10:48.000931 2664 leader_election.cc:288] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Requested vote from peers 780719235a2447119eff68a541837593 (127.0.97.2:40269), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:48.000986 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe104e28012246e897effd6374e9922c" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31"
I20260928 05:10:48.001221 2587 raft_consensus.cc:3037] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:48.002765 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.990864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.97.2:36193)
0928 05:10:47.992014 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:47.992042 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:47.992222 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:47.992745 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:47.992752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:47.992779 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:47.993021 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.993033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.994178 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.994181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.997359 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:47.997372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:47.999204 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:47.999209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:47.999401 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:47.999433 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:10:48.000344 (+ 911us) client_negotiation.cc:240] Negotiation successful
0928 05:10:48.000485 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":3,"thread_start_us":896,"threads_started":1}
I20260928 05:10:48.002883 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe104e28012246e897effd6374e9922c" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593"
I20260928 05:10:48.003125 2452 raft_consensus.cc:3037] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:48.003494 2587 raft_consensus.cc:2454] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:48.003940 2241 leader_election.cc:302] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:48.004530 2664 raft_consensus.cc:2781] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:48.005303 2664 raft_consensus.cc:686] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Becoming Leader. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Running, Role: LEADER
I20260928 05:10:48.005632 2664 consensus_queue.cc:237] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.007591 2667 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: TxnStatusTablet state changed. Reason: New leader d8c30be52ff3428385316a0b16af9d1f. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.007917 2667 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:10:48.007686 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe104e28012246e897effd6374e9922c" candidate_uuid: "780719235a2447119eff68a541837593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c30be52ff3428385316a0b16af9d1f" is_pre_election: true
I20260928 05:10:48.010207 2152 catalog_manager.cc:5495] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8c30be52ff3428385316a0b16af9d1f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } health_report { overall_health: UNKNOWN } } }
I20260928 05:10:48.010813 2452 raft_consensus.cc:2454] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:48.013574 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.992164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37721 (local address 127.0.97.2:58315)
0928 05:10:48.003181 (+ 11017us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:48.003203 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:48.003339 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:48.003975 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:48.003981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:48.004014 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:48.004476 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:48.004487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.006015 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.006020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:48.009362 (+ 3342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:48.009375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.012403 (+ 3028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.012413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:48.012680 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.012743 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 05:10:48.012874 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 05:10:48.013072 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10861,"thread_start_us":14520,"threads_started":1}
I20260928 05:10:48.014503 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:47.992499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58315 (local address 127.0.97.3:37721)
0928 05:10:47.999829 (+ 7330us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:47.999839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:48.003531 (+ 3692us) server_negotiation.cc:407] Connection header received
0928 05:10:48.003604 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:48.003610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:48.003692 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:48.003808 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:48.004690 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.005851 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.010749 (+ 4898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.012145 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.012289 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.012387 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:48.013902 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0928 05:10:48.014106 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7148,"thread_start_us":7034,"threads_started":1}
I20260928 05:10:48.015862 2690 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:10:48.016157 2690 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:10:48.016384 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe104e28012246e897effd6374e9922c" candidate_uuid: "780719235a2447119eff68a541837593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31" is_pre_election: true
I20260928 05:10:48.016697 2587 raft_consensus.cc:2379] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 780719235a2447119eff68a541837593 in current term 1: Already voted for candidate d8c30be52ff3428385316a0b16af9d1f in this term.
I20260928 05:10:48.017300 2381 leader_election.cc:302] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 780719235a2447119eff68a541837593; no voters: 45a665d368f54cf79494f6ead7d38f31, d8c30be52ff3428385316a0b16af9d1f
I20260928 05:10:48.017714 2662 raft_consensus.cc:2726] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:10:48.063642 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:48.057561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.0.97.1:43827)
0928 05:10:48.057759 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:48.057764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:48.057780 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:10:48.057968 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:48.057972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:48.058045 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:48.058154 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:48.058614 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.059444 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.061497 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.061911 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.061999 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.062589 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:48.063133 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:48.063237 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:48.063392 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 05:10:48.063450 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:10:48.063920 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:48.057286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.0.1:52426)
0928 05:10:48.057553 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:48.057575 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:48.057748 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:48.058208 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:48.058214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:48.058228 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:48.058443 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:48.058454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.059586 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.059593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:48.061357 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:48.061373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.062059 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.062067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:48.062298 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.062371 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:48.063242 (+ 871us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:48.063265 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:10:48.063367 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 05:10:48.063524 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 05:10:48.069948 2452 raft_consensus.cc:1261] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Refusing update from remote peer d8c30be52ff3428385316a0b16af9d1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:48.070207 2587 raft_consensus.cc:1261] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Refusing update from remote peer d8c30be52ff3428385316a0b16af9d1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:48.070761 2664 consensus_queue.cc:1035] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:48.071043 2667 consensus_queue.cc:1035] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:48.079919 2663 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: TxnStatusTablet state changed. Reason: New leader d8c30be52ff3428385316a0b16af9d1f. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.080206 2663 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:48.082391 2662 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: TxnStatusTablet state changed. Reason: New leader d8c30be52ff3428385316a0b16af9d1f. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.082700 2662 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:48.084553 2680 mvcc.cc:205] Tried to move back new op lower bound from 7334183928087973888 to 7334183927834025984. Current Snapshot: MvccSnapshot[applied={T|T < 7334183928087973888}]
I20260928 05:10:48.084326 2689 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.084718 2689 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:10:48.084966 2689 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.085227 2689 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P d8c30be52ff3428385316a0b16af9d1f: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:10:48.088290 2663 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.088536 2663 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P 45a665d368f54cf79494f6ead7d38f31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:48.090740 2662 tablet_replica.cc:439] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } } }
I20260928 05:10:48.090991 2662 tablet_replica.cc:442] T fe104e28012246e897effd6374e9922c P 780719235a2447119eff68a541837593: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:10:48.103863 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:48.095095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.0.97.1:43827)
0928 05:10:48.095244 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:48.095249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:48.098222 (+ 2973us) server_negotiation.cc:407] Connection header received
0928 05:10:48.098454 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:48.098460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:48.098570 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:48.098686 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:48.099169 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.099903 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.102046 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.102471 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.102645 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.103158 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:48.103272 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:48.103336 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:48.103509 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 05:10:48.103568 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:10:48.124168 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:48.117398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50735 (local address 127.0.97.1:43827)
0928 05:10:48.117543 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:48.117547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:48.119049 (+ 1502us) server_negotiation.cc:407] Connection header received
0928 05:10:48.119248 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:48.119252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:48.119325 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:48.119414 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:48.119862 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.120588 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.122191 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.122928 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.123074 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.123442 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:48.123543 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:48.123640 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:48.123771 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 05:10:48.123830 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260928 05:10:48.124233 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:48.117328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.97.1:50735)
0928 05:10:48.118948 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:48.118963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:48.119131 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:48.119435 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:48.119439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:48.119448 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:48.119731 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:48.119739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.120721 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.120729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:48.122061 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:48.122076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:48.123090 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:48.123096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:48.123218 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:48.123264 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:48.123636 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:48.123646 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:10:48.123755 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 05:10:48.123884 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"thread_start_us":1357,"threads_started":1}
I20260928 05:10:48.201433 2152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:48.201763 2152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:48.205991 2152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Sending DeleteTablet for 3 replicas of tablet 764e3c36cc47420da635413eac70748e
I20260928 05:10:48.207080 2567 tablet_service.cc:1514] Processing DeleteTablet for tablet 764e3c36cc47420da635413eac70748e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46246
I20260928 05:10:48.208113 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 764e3c36cc47420da635413eac70748e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38154
I20260928 05:10:48.209218 2289 tablet_service.cc:1514] Processing DeleteTablet for tablet 764e3c36cc47420da635413eac70748e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52418
I20260928 05:10:48.210095 2706 tablet_replica.cc:330] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: stopping tablet replica
I20260928 05:10:48.211054 2705 tablet_replica.cc:330] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: stopping tablet replica
I20260928 05:10:48.211493 2152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:10:48.211663 2705 raft_consensus.cc:2229] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:48.212253 2705 raft_consensus.cc:2258] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 05:10:48.212297 2152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:48.214409 2706 raft_consensus.cc:2229] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:10:48.214493 2705 ts_tablet_manager.cc:1905] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:48.215104 2706 raft_consensus.cc:2258] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:48.216809 2706 ts_tablet_manager.cc:1905] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:48.220340 2289 tablet_service.cc:1467] Processing CreateTablet for tablet 98b46e189a234b1e89eb53ca57f1a0bd (DEFAULT_TABLE table=test-table [id=df10bdacb7364f069da45e6c762b02e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:48.220873 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b46e189a234b1e89eb53ca57f1a0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:48.222419 2705 ts_tablet_manager.cc:1918] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:48.222590 2705 log.cc:1196] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/764e3c36cc47420da635413eac70748e
I20260928 05:10:48.222939 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 98b46e189a234b1e89eb53ca57f1a0bd (DEFAULT_TABLE table=test-table [id=df10bdacb7364f069da45e6c762b02e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:48.223165 2705 ts_tablet_manager.cc:1939] T 764e3c36cc47420da635413eac70748e P 780719235a2447119eff68a541837593: Deleting consensus metadata
I20260928 05:10:48.223343 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b46e189a234b1e89eb53ca57f1a0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:48.224452 2706 ts_tablet_manager.cc:1918] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:48.224593 2706 log.cc:1196] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/764e3c36cc47420da635413eac70748e
I20260928 05:10:48.225097 2706 ts_tablet_manager.cc:1939] T 764e3c36cc47420da635413eac70748e P d8c30be52ff3428385316a0b16af9d1f: Deleting consensus metadata
I20260928 05:10:48.226181 2136 catalog_manager.cc:4841] TS 780719235a2447119eff68a541837593 (127.0.97.2:40269): tablet 764e3c36cc47420da635413eac70748e (table test-table [id=25d20a73db2b4b54b01f04826cac5320]) successfully deleted
I20260928 05:10:48.226644 2138 catalog_manager.cc:4841] TS d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827): tablet 764e3c36cc47420da635413eac70748e (table test-table [id=25d20a73db2b4b54b01f04826cac5320]) successfully deleted
I20260928 05:10:48.227169 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 98b46e189a234b1e89eb53ca57f1a0bd (DEFAULT_TABLE table=test-table [id=df10bdacb7364f069da45e6c762b02e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:48.227702 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b46e189a234b1e89eb53ca57f1a0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:48.237021 2708 tablet_bootstrap.cc:492] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Bootstrap starting.
I20260928 05:10:48.239490 2707 tablet_bootstrap.cc:492] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Bootstrap starting.
I20260928 05:10:48.240442 2709 tablet_bootstrap.cc:492] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Bootstrap starting.
I20260928 05:10:48.241498 2708 tablet_bootstrap.cc:654] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:48.242285 2707 tablet_bootstrap.cc:654] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:48.246035 2709 tablet_bootstrap.cc:654] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:48.248709 2704 tablet_replica.cc:330] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: stopping tablet replica
I20260928 05:10:48.249141 2704 raft_consensus.cc:2229] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:48.249532 2704 raft_consensus.cc:2258] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:48.251250 2708 tablet_bootstrap.cc:492] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: No bootstrap required, opened a new log
I20260928 05:10:48.251451 2704 ts_tablet_manager.cc:1905] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:48.251502 2708 ts_tablet_manager.cc:1397] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260928 05:10:48.252699 2708 raft_consensus.cc:348] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.253149 2708 raft_consensus.cc:374] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:48.253325 2708 raft_consensus.cc:729] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Initialized, Role: FOLLOWER
I20260928 05:10:48.253746 2708 consensus_queue.cc:260] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.255501 2709 tablet_bootstrap.cc:492] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: No bootstrap required, opened a new log
I20260928 05:10:48.255722 2709 ts_tablet_manager.cc:1397] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20260928 05:10:48.256983 2709 raft_consensus.cc:348] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.257824 2708 ts_tablet_manager.cc:1428] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 05:10:48.258569 2707 tablet_bootstrap.cc:492] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: No bootstrap required, opened a new log
I20260928 05:10:48.258803 2707 ts_tablet_manager.cc:1397] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.007s
I20260928 05:10:48.259680 2707 raft_consensus.cc:348] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.260026 2707 raft_consensus.cc:374] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:48.260183 2707 raft_consensus.cc:729] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780719235a2447119eff68a541837593, State: Initialized, Role: FOLLOWER
I20260928 05:10:48.260529 2707 consensus_queue.cc:260] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.261245 2707 ts_tablet_manager.cc:1428] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:10:48.260797 2704 ts_tablet_manager.cc:1918] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:48.261463 2709 raft_consensus.cc:374] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:48.261600 2704 log.cc:1196] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/764e3c36cc47420da635413eac70748e
I20260928 05:10:48.261659 2709 raft_consensus.cc:729] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a665d368f54cf79494f6ead7d38f31, State: Initialized, Role: FOLLOWER
I20260928 05:10:48.262197 2704 ts_tablet_manager.cc:1939] T 764e3c36cc47420da635413eac70748e P 45a665d368f54cf79494f6ead7d38f31: Deleting consensus metadata
I20260928 05:10:48.262085 2709 consensus_queue.cc:260] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.263949 2138 catalog_manager.cc:4841] TS 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721): tablet 764e3c36cc47420da635413eac70748e (table test-table [id=25d20a73db2b4b54b01f04826cac5320]) successfully deleted
I20260928 05:10:48.264886 2709 ts_tablet_manager.cc:1428] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260928 05:10:48.622401 2689 raft_consensus.cc:482] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:48.622689 2689 raft_consensus.cc:504] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.623420 2689 leader_election.cc:288] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780719235a2447119eff68a541837593 (127.0.97.2:40269), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:48.623927 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b46e189a234b1e89eb53ca57f1a0bd" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31" is_pre_election: true
I20260928 05:10:48.623934 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b46e189a234b1e89eb53ca57f1a0bd" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593" is_pre_election: true
I20260928 05:10:48.624151 2587 raft_consensus.cc:2454] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:48.624166 2452 raft_consensus.cc:2454] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:48.624637 2239 leader_election.cc:302] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780719235a2447119eff68a541837593, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:48.624904 2689 raft_consensus.cc:2781] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:48.625036 2689 raft_consensus.cc:482] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:48.625150 2689 raft_consensus.cc:3037] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:48.627404 2689 raft_consensus.cc:504] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.627925 2689 leader_election.cc:288] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Requested vote from peers 780719235a2447119eff68a541837593 (127.0.97.2:40269), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:48.628326 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b46e189a234b1e89eb53ca57f1a0bd" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593"
I20260928 05:10:48.628433 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b46e189a234b1e89eb53ca57f1a0bd" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31"
I20260928 05:10:48.628547 2452 raft_consensus.cc:3037] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:48.628630 2587 raft_consensus.cc:3037] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:48.631389 2587 raft_consensus.cc:2454] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:48.631417 2452 raft_consensus.cc:2454] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:48.631760 2239 leader_election.cc:302] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780719235a2447119eff68a541837593, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:48.632050 2689 raft_consensus.cc:2781] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:48.632210 2689 raft_consensus.cc:686] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Becoming Leader. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Running, Role: LEADER
I20260928 05:10:48.632503 2689 consensus_queue.cc:237] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:48.635351 2150 catalog_manager.cc:5495] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8c30be52ff3428385316a0b16af9d1f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } health_report { overall_health: UNKNOWN } } }
I20260928 05:10:48.711854 2587 raft_consensus.cc:1261] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Refusing update from remote peer d8c30be52ff3428385316a0b16af9d1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:48.712163 2453 raft_consensus.cc:1261] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Refusing update from remote peer d8c30be52ff3428385316a0b16af9d1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:48.712694 2689 consensus_queue.cc:1035] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:48.712980 2667 consensus_queue.cc:1035] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:10:48.739130 2714 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 05:10:50.333889 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:50.334151 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:50.337798 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Sending DeleteTablet for 3 replicas of tablet 98b46e189a234b1e89eb53ca57f1a0bd
I20260928 05:10:50.338480 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b46e189a234b1e89eb53ca57f1a0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52418
I20260928 05:10:50.338644 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b46e189a234b1e89eb53ca57f1a0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38154
I20260928 05:10:50.338696 2567 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b46e189a234b1e89eb53ca57f1a0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46246
I20260928 05:10:50.348675 2746 tablet_replica.cc:330] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: stopping tablet replica
I20260928 05:10:50.348855 2747 tablet_replica.cc:330] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: stopping tablet replica
I20260928 05:10:50.348975 2746 raft_consensus.cc:2229] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:10:50.349033 2748 tablet_replica.cc:330] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: stopping tablet replica
I20260928 05:10:50.349156 2747 raft_consensus.cc:2229] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:50.349409 2746 raft_consensus.cc:2258] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:50.349649 2748 raft_consensus.cc:2229] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:50.349867 2747 raft_consensus.cc:2258] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:50.349920 2748 raft_consensus.cc:2258] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:50.350320 2746 ts_tablet_manager.cc:1905] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:50.350868 2747 ts_tablet_manager.cc:1905] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:50.350868 2748 ts_tablet_manager.cc:1905] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:50.357321 2748 ts_tablet_manager.cc:1918] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:10:50.357321 2747 ts_tablet_manager.cc:1918] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:10:50.357468 2747 log.cc:1196] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/98b46e189a234b1e89eb53ca57f1a0bd
I20260928 05:10:50.357501 2748 log.cc:1196] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/98b46e189a234b1e89eb53ca57f1a0bd
I20260928 05:10:50.357932 2746 ts_tablet_manager.cc:1918] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:10:50.357971 2747 ts_tablet_manager.cc:1939] T 98b46e189a234b1e89eb53ca57f1a0bd P 780719235a2447119eff68a541837593: Deleting consensus metadata
I20260928 05:10:50.357973 2748 ts_tablet_manager.cc:1939] T 98b46e189a234b1e89eb53ca57f1a0bd P 45a665d368f54cf79494f6ead7d38f31: Deleting consensus metadata
I20260928 05:10:50.358140 2746 log.cc:1196] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/98b46e189a234b1e89eb53ca57f1a0bd
I20260928 05:10:50.358677 2746 ts_tablet_manager.cc:1939] T 98b46e189a234b1e89eb53ca57f1a0bd P d8c30be52ff3428385316a0b16af9d1f: Deleting consensus metadata
I20260928 05:10:50.359076 2138 catalog_manager.cc:4841] TS 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721): tablet 98b46e189a234b1e89eb53ca57f1a0bd (table test-table [id=df10bdacb7364f069da45e6c762b02e7]) successfully deleted
I20260928 05:10:50.359319 2136 catalog_manager.cc:4841] TS 780719235a2447119eff68a541837593 (127.0.97.2:40269): tablet 98b46e189a234b1e89eb53ca57f1a0bd (table test-table [id=df10bdacb7364f069da45e6c762b02e7]) successfully deleted
I20260928 05:10:50.359858 2138 catalog_manager.cc:4841] TS d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827): tablet 98b46e189a234b1e89eb53ca57f1a0bd (table test-table [id=df10bdacb7364f069da45e6c762b02e7]) successfully deleted
Sep 28 05:10:50 dist-test-slave-6718 krb5kdc[2107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:10:50 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:50.391227 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.377293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.97.62:45517)
0928 05:10:50.377863 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.377872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.377974 (+ 102us) server_negotiation.cc:407] Connection header received
0928 05:10:50.378195 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.378198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.378264 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.378372 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:50.380174 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.381061 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.382296 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.382742 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.382889 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.386576 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:50.386585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:50.386592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:50.386630 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:50.389031 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:50.389642 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:50.389645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:50.389648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:50.389757 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:50.390053 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:50.390056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:50.390058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:50.390364 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:50.390615 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.390834 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.390982 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":255,"threads_started":1}
I20260928 05:10:50.394315 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:10:50.394896 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:50.400844 2288 tablet_service.cc:1467] Processing CreateTablet for tablet f71a3c8c590c4f4ea8117ed1f8e56b10 (DEFAULT_TABLE table=test-table [id=8f9205fc0eaf4cdeb3847002c87399b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:50.400980 2428 tablet_service.cc:1467] Processing CreateTablet for tablet f71a3c8c590c4f4ea8117ed1f8e56b10 (DEFAULT_TABLE table=test-table [id=8f9205fc0eaf4cdeb3847002c87399b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:50.401330 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71a3c8c590c4f4ea8117ed1f8e56b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:50.401357 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71a3c8c590c4f4ea8117ed1f8e56b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:50.401331 2567 tablet_service.cc:1467] Processing CreateTablet for tablet f71a3c8c590c4f4ea8117ed1f8e56b10 (DEFAULT_TABLE table=test-table [id=8f9205fc0eaf4cdeb3847002c87399b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:50.401754 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71a3c8c590c4f4ea8117ed1f8e56b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:50.407826 2757 tablet_bootstrap.cc:492] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Bootstrap starting.
I20260928 05:10:50.409125 2757 tablet_bootstrap.cc:654] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:50.411715 2759 tablet_bootstrap.cc:492] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Bootstrap starting.
I20260928 05:10:50.412297 2757 tablet_bootstrap.cc:492] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: No bootstrap required, opened a new log
I20260928 05:10:50.412436 2757 ts_tablet_manager.cc:1397] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 05:10:50.413046 2757 raft_consensus.cc:348] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.413213 2757 raft_consensus.cc:374] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:50.413362 2757 raft_consensus.cc:729] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a665d368f54cf79494f6ead7d38f31, State: Initialized, Role: FOLLOWER
I20260928 05:10:50.413616 2757 consensus_queue.cc:260] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.414897 2757 ts_tablet_manager.cc:1428] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:10:50.415971 2758 tablet_bootstrap.cc:492] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Bootstrap starting.
I20260928 05:10:50.416443 2759 tablet_bootstrap.cc:654] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:50.417438 2758 tablet_bootstrap.cc:654] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:50.418861 2759 tablet_bootstrap.cc:492] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: No bootstrap required, opened a new log
I20260928 05:10:50.419011 2759 ts_tablet_manager.cc:1397] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 05:10:50.419600 2758 tablet_bootstrap.cc:492] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: No bootstrap required, opened a new log
I20260928 05:10:50.419754 2758 ts_tablet_manager.cc:1397] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:10:50.419766 2759 raft_consensus.cc:348] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.420058 2759 raft_consensus.cc:374] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:50.420159 2759 raft_consensus.cc:729] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780719235a2447119eff68a541837593, State: Initialized, Role: FOLLOWER
I20260928 05:10:50.420385 2759 consensus_queue.cc:260] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.420517 2758 raft_consensus.cc:348] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.420802 2758 raft_consensus.cc:374] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:50.420920 2758 raft_consensus.cc:729] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Initialized, Role: FOLLOWER
I20260928 05:10:50.421231 2759 ts_tablet_manager.cc:1428] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:10:50.421152 2758 consensus_queue.cc:260] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.421772 2758 ts_tablet_manager.cc:1428] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:10:50.505189 2761 raft_consensus.cc:482] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:50.505482 2761 raft_consensus.cc:504] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.506953 2761 leader_election.cc:288] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827), 780719235a2447119eff68a541837593 (127.0.97.2:40269)
I20260928 05:10:50.523716 2764 raft_consensus.cc:482] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:50.523949 2764 raft_consensus.cc:504] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.524643 2764 leader_election.cc:288] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:50.525233 2586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "780719235a2447119eff68a541837593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31" is_pre_election: true
I20260928 05:10:50.525501 2586 raft_consensus.cc:2454] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 780719235a2447119eff68a541837593 in term 0.
I20260928 05:10:50.525697 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "780719235a2447119eff68a541837593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c30be52ff3428385316a0b16af9d1f" is_pre_election: true
I20260928 05:10:50.526027 2309 raft_consensus.cc:2454] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 780719235a2447119eff68a541837593 in term 0.
I20260928 05:10:50.526048 2381 leader_election.cc:302] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31, 780719235a2447119eff68a541837593; no voters:
I20260928 05:10:50.526448 2764 raft_consensus.cc:2781] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:50.526613 2764 raft_consensus.cc:482] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:50.526701 2764 raft_consensus.cc:3037] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:50.527799 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.507341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42943 (local address 127.0.97.2:40269)
0928 05:10:50.519729 (+ 12388us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.519736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.520362 (+ 626us) server_negotiation.cc:407] Connection header received
0928 05:10:50.520507 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.520512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.520580 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.520683 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:50.521345 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.522479 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.525152 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.526771 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.526931 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.527049 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.527299 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.527433 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12233,"thread_start_us":12740,"threads_started":1}
I20260928 05:10:50.527812 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.507235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40269 (local address 127.0.97.3:42943)
0928 05:10:50.520246 (+ 13011us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:50.520262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:50.520387 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:50.520710 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:50.520714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:50.520732 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:50.521172 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.521180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.522676 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.522679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.524944 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.524958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.526976 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.526982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.527148 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.527170 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:10:50.527278 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:10:50.527411 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12875,"mutex_wait_us":105,"thread_start_us":12162,"threads_started":1}
I20260928 05:10:50.528266 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.507232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.97.3:59101)
0928 05:10:50.519732 (+ 12500us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:50.519755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:50.519968 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:50.520541 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:50.520544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:50.520562 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:10:50.520941 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.520952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.522669 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.522672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.524960 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.524972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.527324 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.527331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.527497 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.527527 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:10:50.527611 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 05:10:50.527744 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12355,"mutex_wait_us":149,"thread_start_us":12661,"threads_started":1}
I20260928 05:10:50.528308 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.507341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59101 (local address 127.0.97.1:43827)
0928 05:10:50.520178 (+ 12837us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.520184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.520207 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:10:50.520274 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.520278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.520368 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.520508 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:10:50.521144 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.522446 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.525137 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.526950 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.527071 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.527192 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.527873 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.528013 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12666,"thread_start_us":12753,"threads_started":1}
I20260928 05:10:50.528426 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "45a665d368f54cf79494f6ead7d38f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593" is_pre_election: true
I20260928 05:10:50.528753 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "45a665d368f54cf79494f6ead7d38f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c30be52ff3428385316a0b16af9d1f" is_pre_election: true
I20260928 05:10:50.528950 2309 raft_consensus.cc:2454] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a665d368f54cf79494f6ead7d38f31 in term 0.
I20260928 05:10:50.529351 2520 leader_election.cc:302] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:50.529704 2761 raft_consensus.cc:2781] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:50.529827 2761 raft_consensus.cc:482] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:50.529930 2761 raft_consensus.cc:3037] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:50.529810 2764 raft_consensus.cc:504] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.530220 2453 raft_consensus.cc:2379] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45a665d368f54cf79494f6ead7d38f31 in current term 1: Already voted for candidate 780719235a2447119eff68a541837593 in this term.
I20260928 05:10:50.530515 2764 leader_election.cc:288] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [CANDIDATE]: Term 1 election: Requested vote from peers d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:50.530925 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "780719235a2447119eff68a541837593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c30be52ff3428385316a0b16af9d1f"
I20260928 05:10:50.530968 2586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "780719235a2447119eff68a541837593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31"
I20260928 05:10:50.531107 2309 raft_consensus.cc:3037] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:50.532385 2761 raft_consensus.cc:504] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.532702 2586 raft_consensus.cc:2379] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 780719235a2447119eff68a541837593 in current term 1: Already voted for candidate 45a665d368f54cf79494f6ead7d38f31 in this term.
I20260928 05:10:50.533075 2761 leader_election.cc:288] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [CANDIDATE]: Term 1 election: Requested vote from peers d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827), 780719235a2447119eff68a541837593 (127.0.97.2:40269)
I20260928 05:10:50.533131 2309 raft_consensus.cc:2454] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 780719235a2447119eff68a541837593 in term 1.
I20260928 05:10:50.533423 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "45a665d368f54cf79494f6ead7d38f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c30be52ff3428385316a0b16af9d1f"
I20260928 05:10:50.533627 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71a3c8c590c4f4ea8117ed1f8e56b10" candidate_uuid: "45a665d368f54cf79494f6ead7d38f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593"
I20260928 05:10:50.533721 2308 raft_consensus.cc:2379] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 45a665d368f54cf79494f6ead7d38f31 in current term 1: Already voted for candidate 780719235a2447119eff68a541837593 in this term.
I20260928 05:10:50.533877 2453 raft_consensus.cc:2379] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 45a665d368f54cf79494f6ead7d38f31 in current term 1: Already voted for candidate 780719235a2447119eff68a541837593 in this term.
I20260928 05:10:50.534610 2381 leader_election.cc:302] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 780719235a2447119eff68a541837593, d8c30be52ff3428385316a0b16af9d1f; no voters: 45a665d368f54cf79494f6ead7d38f31
I20260928 05:10:50.534619 2518 leader_election.cc:302] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45a665d368f54cf79494f6ead7d38f31; no voters: 780719235a2447119eff68a541837593, d8c30be52ff3428385316a0b16af9d1f
I20260928 05:10:50.534920 2761 raft_consensus.cc:2726] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 05:10:50.534883 2764 raft_consensus.cc:2781] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:50.535328 2764 raft_consensus.cc:686] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 LEADER]: Becoming Leader. State: Replica: 780719235a2447119eff68a541837593, State: Running, Role: LEADER
I20260928 05:10:50.535750 2764 consensus_queue.cc:237] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.538995 2150 catalog_manager.cc:5495] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 780719235a2447119eff68a541837593 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "780719235a2447119eff68a541837593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } health_report { overall_health: UNKNOWN } } }
I20260928 05:10:50.570866 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.565708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.97.2:40269)
0928 05:10:50.565932 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.565938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.566055 (+ 117us) server_negotiation.cc:407] Connection header received
0928 05:10:50.566229 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.566233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.566303 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.566437 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:50.566841 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.567512 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.568875 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.569250 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.569336 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.569767 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:50.570310 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:50.570476 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.570524 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.570621 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 05:10:50.585793 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.580689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39965 (local address 127.0.97.1:43827)
0928 05:10:50.580919 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.580927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.581294 (+ 367us) server_negotiation.cc:407] Connection header received
0928 05:10:50.581499 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.581504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.581575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.581668 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:50.582097 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.582865 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.584159 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.584567 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.584707 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.585093 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:50.585186 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:50.585280 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.585410 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.585501 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:10:50.585879 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.580667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.97.2:39965)
0928 05:10:50.581192 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:50.581207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:50.581365 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:50.581683 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:50.581687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:50.581698 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:50.581962 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.581971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.582995 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.583001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.584021 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.584032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.584739 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.584746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.584873 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.584924 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:50.585280 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:50.585293 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 05:10:50.585390 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:10:50.585521 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":235,"threads_started":1}
I20260928 05:10:50.594162 2586 raft_consensus.cc:1261] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Refusing update from remote peer 780719235a2447119eff68a541837593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:50.594461 2309 raft_consensus.cc:1261] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Refusing update from remote peer 780719235a2447119eff68a541837593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:50.594866 2769 consensus_queue.cc:1035] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:50.595279 2764 consensus_queue.cc:1035] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:50.622242 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.616800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55185 (local address 127.0.97.2:40269)
0928 05:10:50.616972 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.616976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.617445 (+ 469us) server_negotiation.cc:407] Connection header received
0928 05:10:50.617577 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.617580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.617636 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.617705 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:50.618143 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.619130 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.620586 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.620969 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.621069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.621525 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:50.621651 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:50.621735 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.621939 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.622018 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:10:50.622428 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.616757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40269 (local address 127.0.97.1:55185)
0928 05:10:50.617360 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:50.617375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:50.617481 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:50.617741 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:50.617744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:50.617753 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:10:50.618004 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.618011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.619270 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.619274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.620467 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:50.620480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.621125 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.621130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:50.621293 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.621347 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:10:50.621786 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:10:50.621799 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 05:10:50.621887 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 05:10:50.622060 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
I20260928 05:10:50.652781 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:50.653043 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:50.655982 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Sending DeleteTablet for 3 replicas of tablet f71a3c8c590c4f4ea8117ed1f8e56b10
I20260928 05:10:50.656694 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet f71a3c8c590c4f4ea8117ed1f8e56b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52418
I20260928 05:10:50.656912 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet f71a3c8c590c4f4ea8117ed1f8e56b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38154
I20260928 05:10:50.656970 2746 tablet_replica.cc:330] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: stopping tablet replica
I20260928 05:10:50.656975 2567 tablet_service.cc:1514] Processing DeleteTablet for tablet f71a3c8c590c4f4ea8117ed1f8e56b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46246
I20260928 05:10:50.657169 2746 raft_consensus.cc:2229] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:50.657241 2747 tablet_replica.cc:330] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: stopping tablet replica
I20260928 05:10:50.657366 2746 raft_consensus.cc:2258] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:50.657429 2748 tablet_replica.cc:330] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: stopping tablet replica
I20260928 05:10:50.657426 2747 raft_consensus.cc:2229] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:10:50.657622 2748 raft_consensus.cc:2229] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:50.657817 2748 raft_consensus.cc:2258] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:50.657883 2747 raft_consensus.cc:2258] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:50.658658 2746 ts_tablet_manager.cc:1905] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:50.659061 2747 ts_tablet_manager.cc:1905] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:50.663458 2748 ts_tablet_manager.cc:1905] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:50.666219 2746 ts_tablet_manager.cc:1918] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:50.666225 2747 ts_tablet_manager.cc:1918] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:50.666370 2746 log.cc:1196] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/f71a3c8c590c4f4ea8117ed1f8e56b10
I20260928 05:10:50.666395 2747 log.cc:1196] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/f71a3c8c590c4f4ea8117ed1f8e56b10
I20260928 05:10:50.666458 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:10:50.666843 2746 ts_tablet_manager.cc:1939] T f71a3c8c590c4f4ea8117ed1f8e56b10 P d8c30be52ff3428385316a0b16af9d1f: Deleting consensus metadata
I20260928 05:10:50.666859 2747 ts_tablet_manager.cc:1939] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 780719235a2447119eff68a541837593: Deleting consensus metadata
W20260928 05:10:50.667165 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:10:50.667882 2138 catalog_manager.cc:4841] TS d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827): tablet f71a3c8c590c4f4ea8117ed1f8e56b10 (table test-table [id=8f9205fc0eaf4cdeb3847002c87399b7]) successfully deleted
I20260928 05:10:50.668116 2136 catalog_manager.cc:4841] TS 780719235a2447119eff68a541837593 (127.0.97.2:40269): tablet f71a3c8c590c4f4ea8117ed1f8e56b10 (table test-table [id=8f9205fc0eaf4cdeb3847002c87399b7]) successfully deleted
I20260928 05:10:50.672343 2748 ts_tablet_manager.cc:1918] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:10:50.672482 2748 log.cc:1196] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/f71a3c8c590c4f4ea8117ed1f8e56b10
I20260928 05:10:50.672875 2748 ts_tablet_manager.cc:1939] T f71a3c8c590c4f4ea8117ed1f8e56b10 P 45a665d368f54cf79494f6ead7d38f31: Deleting consensus metadata
I20260928 05:10:50.674671 2138 catalog_manager.cc:4841] TS 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721): tablet f71a3c8c590c4f4ea8117ed1f8e56b10 (table test-table [id=8f9205fc0eaf4cdeb3847002c87399b7]) successfully deleted
I20260928 05:10:50.674932 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 0a685f2109244f8bbc86f4d209a2b2a3 (DEFAULT_TABLE table=test-table [id=89a02ee2616140159219446e81aa9f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:50.675032 2288 tablet_service.cc:1467] Processing CreateTablet for tablet 0a685f2109244f8bbc86f4d209a2b2a3 (DEFAULT_TABLE table=test-table [id=89a02ee2616140159219446e81aa9f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:50.675364 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a685f2109244f8bbc86f4d209a2b2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:50.675374 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a685f2109244f8bbc86f4d209a2b2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:50.679481 2567 tablet_service.cc:1467] Processing CreateTablet for tablet 0a685f2109244f8bbc86f4d209a2b2a3 (DEFAULT_TABLE table=test-table [id=89a02ee2616140159219446e81aa9f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:10:50.679883 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a685f2109244f8bbc86f4d209a2b2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:50.685487 2758 tablet_bootstrap.cc:492] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: Bootstrap starting.
I20260928 05:10:50.687000 2758 tablet_bootstrap.cc:654] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:50.690294 2759 tablet_bootstrap.cc:492] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Bootstrap starting.
I20260928 05:10:50.690737 2757 tablet_bootstrap.cc:492] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Bootstrap starting.
I20260928 05:10:50.692124 2759 tablet_bootstrap.cc:654] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:50.692346 2757 tablet_bootstrap.cc:654] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:50.698678 2758 tablet_bootstrap.cc:492] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: No bootstrap required, opened a new log
I20260928 05:10:50.698863 2758 ts_tablet_manager.cc:1397] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260928 05:10:50.699682 2758 raft_consensus.cc:348] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.699951 2758 raft_consensus.cc:374] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:50.700045 2758 raft_consensus.cc:729] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Initialized, Role: FOLLOWER
I20260928 05:10:50.700307 2758 consensus_queue.cc:260] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.701063 2759 tablet_bootstrap.cc:492] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: No bootstrap required, opened a new log
I20260928 05:10:50.701820 2758 ts_tablet_manager.cc:1428] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:10:50.702009 2759 ts_tablet_manager.cc:1397] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260928 05:10:50.703033 2757 tablet_bootstrap.cc:492] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: No bootstrap required, opened a new log
I20260928 05:10:50.703263 2757 ts_tablet_manager.cc:1397] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260928 05:10:50.704083 2757 raft_consensus.cc:348] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.704406 2757 raft_consensus.cc:374] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:50.704535 2757 raft_consensus.cc:729] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a665d368f54cf79494f6ead7d38f31, State: Initialized, Role: FOLLOWER
I20260928 05:10:50.704869 2757 consensus_queue.cc:260] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.707165 2757 ts_tablet_manager.cc:1428] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 05:10:50.708086 2759 raft_consensus.cc:348] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.708352 2759 raft_consensus.cc:374] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:50.708451 2759 raft_consensus.cc:729] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780719235a2447119eff68a541837593, State: Initialized, Role: FOLLOWER
I20260928 05:10:50.708815 2759 consensus_queue.cc:260] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.710004 2759 ts_tablet_manager.cc:1428] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260928 05:10:50.947624 2729 raft_consensus.cc:482] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:10:50.947890 2729 raft_consensus.cc:504] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.948645 2729 leader_election.cc:288] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780719235a2447119eff68a541837593 (127.0.97.2:40269), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:50.949247 2586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a685f2109244f8bbc86f4d209a2b2a3" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31" is_pre_election: true
I20260928 05:10:50.949276 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a685f2109244f8bbc86f4d209a2b2a3" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593" is_pre_election: true
I20260928 05:10:50.949520 2586 raft_consensus.cc:2454] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:50.949572 2453 raft_consensus.cc:2454] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 0.
I20260928 05:10:50.950045 2239 leader_election.cc:302] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780719235a2447119eff68a541837593, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:50.950323 2729 raft_consensus.cc:2781] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:10:50.950466 2729 raft_consensus.cc:482] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:10:50.950614 2729 raft_consensus.cc:3037] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:50.952757 2729 raft_consensus.cc:504] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.953435 2729 leader_election.cc:288] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Requested vote from peers 780719235a2447119eff68a541837593 (127.0.97.2:40269), 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721)
I20260928 05:10:50.953958 2586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a685f2109244f8bbc86f4d209a2b2a3" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a665d368f54cf79494f6ead7d38f31"
I20260928 05:10:50.953994 2453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a685f2109244f8bbc86f4d209a2b2a3" candidate_uuid: "d8c30be52ff3428385316a0b16af9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780719235a2447119eff68a541837593"
I20260928 05:10:50.954191 2586 raft_consensus.cc:3037] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:50.954205 2453 raft_consensus.cc:3037] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:50.956578 2453 raft_consensus.cc:2454] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:50.957067 2239 leader_election.cc:302] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780719235a2447119eff68a541837593, d8c30be52ff3428385316a0b16af9d1f; no voters:
I20260928 05:10:50.957437 2729 raft_consensus.cc:2781] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:50.957613 2729 raft_consensus.cc:686] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Becoming Leader. State: Replica: d8c30be52ff3428385316a0b16af9d1f, State: Running, Role: LEADER
I20260928 05:10:50.957613 2586 raft_consensus.cc:2454] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c30be52ff3428385316a0b16af9d1f in term 1.
I20260928 05:10:50.957918 2729 consensus_queue.cc:237] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } }
I20260928 05:10:50.960971 2152 catalog_manager.cc:5495] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d8c30be52ff3428385316a0b16af9d1f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d8c30be52ff3428385316a0b16af9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c30be52ff3428385316a0b16af9d1f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 } health_report { overall_health: UNKNOWN } } }
I20260928 05:10:50.993599 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:50.988703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.97.1:43827)
0928 05:10:50.988884 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:50.988893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:50.988909 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:10:50.988955 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:50.988959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:50.989015 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:50.989085 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:10:50.989437 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.990406 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.991766 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:50.992183 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:50.992323 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:50.992732 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:10:50.992835 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:10:50.993056 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:50.993226 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 05:10:50.993311 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:10:51.002813 2453 raft_consensus.cc:1261] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Refusing update from remote peer d8c30be52ff3428385316a0b16af9d1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:51.003161 2587 raft_consensus.cc:1261] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Refusing update from remote peer d8c30be52ff3428385316a0b16af9d1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:10:51.003423 2738 consensus_queue.cc:1035] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "780719235a2447119eff68a541837593" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:51.003822 2729 consensus_queue.cc:1035] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a665d368f54cf79494f6ead7d38f31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:10:51.007688 2727 mvcc.cc:205] Tried to move back new op lower bound from 7334183940102242304 to 7334183939926474752. Current Snapshot: MvccSnapshot[applied={T|T < 7334183940102242304}]
W20260928 05:10:51.032385 2727 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 05:10:52.622694 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:52.622972 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:10:52.627915 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1345207eaa2b46e38ae72768434b57a5: Sending DeleteTablet for 3 replicas of tablet 0a685f2109244f8bbc86f4d209a2b2a3
I20260928 05:10:52.628747 2289 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a685f2109244f8bbc86f4d209a2b2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52418
I20260928 05:10:52.629240 2567 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a685f2109244f8bbc86f4d209a2b2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46246
I20260928 05:10:52.629731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2223
I20260928 05:10:52.629781 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a685f2109244f8bbc86f4d209a2b2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:10:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38154
I20260928 05:10:52.635528 2811 tablet_replica.cc:330] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: stopping tablet replica
I20260928 05:10:52.636021 2811 raft_consensus.cc:2229] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:52.636389 2811 raft_consensus.cc:2258] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:52.638406 2811 ts_tablet_manager.cc:1905] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:52.638748 2810 tablet_replica.cc:330] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: stopping tablet replica
I20260928 05:10:52.638952 2812 tablet_replica.cc:330] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: stopping tablet replica
I20260928 05:10:52.639106 2810 raft_consensus.cc:2229] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:10:52.639192 2812 raft_consensus.cc:2229] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:10:52.639503 2812 raft_consensus.cc:2258] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:52.639586 2810 raft_consensus.cc:2258] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:10:52.640504 2812 ts_tablet_manager.cc:1905] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:10:52.641055 2810 ts_tablet_manager.cc:1905] T 0a685f2109244f8bbc86f4d209a2b2a3 P d8c30be52ff3428385316a0b16af9d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:10:52 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:10:52.658640 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.631395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.0.97.1:43827)
0928 05:10:52.637765 (+ 6370us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:52.637772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:52.637798 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:10:52.637875 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:52.637883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:52.637987 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:52.638133 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:52.640458 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:52.641663 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:52.643583 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:52.644153 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:52.644256 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:52.652091 (+ 7835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:52.652108 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:52.652127 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:52.652170 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:52.655589 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:52.656375 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:52.656383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:52.656385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:52.656552 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:52.657008 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:52.657012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:52.657014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:52.657396 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:52.657608 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:52.658032 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 05:10:52.658221 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6212,"thread_start_us":6086,"threads_started":1}
W20260928 05:10:52.659636 2354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52448
W20260928 05:10:52.660282 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2223: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:10:52.660486 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 2223
I20260928 05:10:52.712903 2812 ts_tablet_manager.cc:1918] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:10:52.713112 2812 log.cc:1196] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/0a685f2109244f8bbc86f4d209a2b2a3
I20260928 05:10:52.713294 2811 ts_tablet_manager.cc:1918] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 05:10:52.713503 2811 log.cc:1196] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/0a685f2109244f8bbc86f4d209a2b2a3
I20260928 05:10:52.713631 2812 ts_tablet_manager.cc:1939] T 0a685f2109244f8bbc86f4d209a2b2a3 P 780719235a2447119eff68a541837593: Deleting consensus metadata
I20260928 05:10:52.713999 2811 ts_tablet_manager.cc:1939] T 0a685f2109244f8bbc86f4d209a2b2a3 P 45a665d368f54cf79494f6ead7d38f31: Deleting consensus metadata
W20260928 05:10:52.734488 2138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43827 (error 108)
I20260928 05:10:52.734854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2363
I20260928 05:10:52.736814 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.735796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.0.1:52452)
0928 05:10:52.736421 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:52.736591 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":366,"threads_started":1}
W20260928 05:10:52.737041 2138 catalog_manager.cc:4584] TS d8c30be52ff3428385316a0b16af9d1f (127.0.97.1:43827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a685f2109244f8bbc86f4d209a2b2a3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43827: connect: Connection refused (error 111)
Sep 28 05:10:52 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:10:52.746342 2136 catalog_manager.cc:4841] TS 780719235a2447119eff68a541837593 (127.0.97.2:40269): tablet 0a685f2109244f8bbc86f4d209a2b2a3 (table test-table [id=89a02ee2616140159219446e81aa9f15]) successfully deleted
I20260928 05:10:52.748658 2138 catalog_manager.cc:4841] TS 45a665d368f54cf79494f6ead7d38f31 (127.0.97.3:37721): tablet 0a685f2109244f8bbc86f4d209a2b2a3 (table test-table [id=89a02ee2616140159219446e81aa9f15]) successfully deleted
I20260928 05:10:52.751092 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.735690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.0.97.2:40269)
0928 05:10:52.736218 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:52.736226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:52.736247 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:10:52.736313 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:52.736318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:52.736395 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:52.736513 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:52.737869 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:52.738751 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:52.740115 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:52.740503 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:52.740604 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:52.745281 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:52.745299 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:52.745319 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:52.745365 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:52.748422 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:52.749247 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:52.749254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:52.749256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:52.749406 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:52.749728 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:52.749731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:52.749733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:52.750085 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:52.750312 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:52.750616 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 05:10:52.750800 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":278,"threads_started":1}
W20260928 05:10:52.751825 2493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38170
W20260928 05:10:52.752468 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:10:52.752614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 2363
I20260928 05:10:52.774690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2502
Sep 28 05:10:52 dist-test-slave-6718 krb5kdc[2107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:10:52.791352 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.790730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.0.1:52462)
0928 05:10:52.791040 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:52.791146 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 05:10:52.791559 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.775474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.0.97.3:37721)
0928 05:10:52.775976 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:52.775980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:52.775996 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:10:52.776048 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:52.776052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:52.776156 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:52.776274 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:52.777525 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:52.778361 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:52.779585 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:52.780028 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:52.780139 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:52.784447 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:52.784463 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:52.784481 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:52.784514 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:52.788274 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:52.789068 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:52.789079 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:52.789084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:52.789246 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:52.789630 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:52.789637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:52.789640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:52.790077 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:52.790401 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:52.790922 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 05:10:52.791142 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
W20260928 05:10:52.792351 2632 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46256
I20260928 05:10:52.792579 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.792183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.0.1:52474)
0928 05:10:52.792332 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:52.792418 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 05:10:52.793000 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:10:52.793174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 2502
I20260928 05:10:52.811614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2120
I20260928 05:10:52.812806 2192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:52.933151 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:52.932657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43827 (local address 127.0.0.1:52476)
0928 05:10:52.932877 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:52.932976 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 05:10:52.933357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 2120
2026-09-28T05:10:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9054 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:53 dist-test-slave-6718 krb5kdc[2822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:53 dist-test-slave-6718 krb5kdc[2822](info): set up 2 sockets
Sep 28 05:10:53 dist-test-slave-6718 krb5kdc[2822](info): commencing operation
krb5kdc: starting...
W20260928 05:10:54.018888 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:10:54 dist-test-slave-6718 krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:10:54.214262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43311
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:54.537389 2835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:54.537719 2835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:54.537931 2835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:54.545362 2835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:54.545485 2835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:54.545573 2835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:54.545657 2835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:54.545745 2835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:54.555701 2835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:54.557024 2835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:54.558343 2835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:54.569957 2840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:54.570019 2841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:54.569986 2843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:54.570644 2835 server_base.cc:1034] running on GCE node
I20260928 05:10:54.571393 2835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:54.572777 2835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:54.574115 2835 hybrid_clock.cc:648] HybridClock initialized: now 1790572254574035 us; error 57 us; skew 500 ppm
I20260928 05:10:54.574569 2835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:54 dist-test-slave-6718 krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:54.577571 2835 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:54.579151 2835 webserver.cc:460] Webserver started at http://127.0.97.62:38625/ using document root <none> and password file <none>
I20260928 05:10:54.579831 2835 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:54.579981 2835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:54.580292 2835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:54.583282 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "ccc00b855a044e36a086dd905be52f0f"
format_stamp: "Formatted at 2026-09-28 05:10:54 on dist-test-slave-6718"
server_key: "b5e229e1c439baca5c59cfd96677e08d"
server_key_iv: "f3a708184add988bcc5029ef55a2e9c0"
server_key_version: "encryptionkey@0"
I20260928 05:10:54.584053 2835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "ccc00b855a044e36a086dd905be52f0f"
format_stamp: "Formatted at 2026-09-28 05:10:54 on dist-test-slave-6718"
server_key: "b5e229e1c439baca5c59cfd96677e08d"
server_key_iv: "f3a708184add988bcc5029ef55a2e9c0"
server_key_version: "encryptionkey@0"
I20260928 05:10:54.589107 2835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:10:54.592552 2850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:54.593845 2835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:10:54.594044 2835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "ccc00b855a044e36a086dd905be52f0f"
format_stamp: "Formatted at 2026-09-28 05:10:54 on dist-test-slave-6718"
server_key: "b5e229e1c439baca5c59cfd96677e08d"
server_key_iv: "f3a708184add988bcc5029ef55a2e9c0"
server_key_version: "encryptionkey@0"
I20260928 05:10:54.594234 2835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:54.614508 2835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:54.618441 2835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:54.618750 2835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:54.636790 2835 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43311
I20260928 05:10:54.636821 2901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43311 every 8 connection(s)
I20260928 05:10:54.638420 2835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:10:54.642151 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:54.643834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 2835
I20260928 05:10:54.644055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:10:54.644848 388 external_mini_cluster.cc:1428] Setting key 9fc803cbee1390e07673e5f34c5dcaa7
I20260928 05:10:54.656952 2902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f: Bootstrap starting.
Sep 28 05:10:54 dist-test-slave-6718 krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:54.660805 2902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:54.662029 2902 log.cc:824] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:54.665026 2902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f: No bootstrap required, opened a new log
I20260928 05:10:54.668077 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:54.646995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.0.97.62:43311)
0928 05:10:54.647839 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:54.647850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:54.647893 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:10:54.649460 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:54.649481 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:54.649885 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:54.650222 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:54.652041 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:54.653363 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:54.655260 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:54.655985 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:54.656127 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:54.660653 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:54.660661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:54.660676 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:54.660717 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:54.663913 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:54.664662 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:54.664668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:54.664675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:54.664833 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:54.665234 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:54.665239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:54.665243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:54.665793 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:54.666113 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:54.666571 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 05:10:54.667020 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":421,"threads_started":1}
I20260928 05:10:54.673017 2902 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc00b855a044e36a086dd905be52f0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43311 } }
I20260928 05:10:54.673394 2902 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:54.673506 2902 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc00b855a044e36a086dd905be52f0f, State: Initialized, Role: FOLLOWER
I20260928 05:10:54.674103 2902 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc00b855a044e36a086dd905be52f0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43311 } }
I20260928 05:10:54.674347 2902 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:10:54.674494 2902 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:10:54.674692 2902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:54.677134 2902 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc00b855a044e36a086dd905be52f0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43311 } }
I20260928 05:10:54.677623 2902 leader_election.cc:302] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc00b855a044e36a086dd905be52f0f; no voters:
I20260928 05:10:54.678241 2902 leader_election.cc:288] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:10:54.678393 2907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:54.679103 2907 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [term 1 LEADER]: Becoming Leader. State: Replica: ccc00b855a044e36a086dd905be52f0f, State: Running, Role: LEADER
I20260928 05:10:54.679657 2907 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc00b855a044e36a086dd905be52f0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43311 } }
I20260928 05:10:54.680392 2902 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:10:54.682443 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccc00b855a044e36a086dd905be52f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc00b855a044e36a086dd905be52f0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43311 } } }
I20260928 05:10:54.682865 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:54.683146 2909 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc00b855a044e36a086dd905be52f0f. Latest consensus state: current_term: 1 leader_uuid: "ccc00b855a044e36a086dd905be52f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc00b855a044e36a086dd905be52f0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43311 } } }
I20260928 05:10:54.683430 2909 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:54.687956 2914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:10:54.693531 2914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:10:54.702109 2914 catalog_manager.cc:1347] Generated new cluster ID: 9b89e577ed8f4fc8aeecf04585019f61
I20260928 05:10:54.702231 2914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:10:54.745628 2914 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:10:54.747064 2914 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:10:54.759544 2914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccc00b855a044e36a086dd905be52f0f: Generated new TSK 0
I20260928 05:10:54.760385 2914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:10:54.809888 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43311
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:55.125861 2928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:55.126130 2928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:55.126279 2928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:55.133947 2928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:55.134054 2928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:55.134235 2928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:10:55.143740 2928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:55.144953 2928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:55.146277 2928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:55.158951 2934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:55.158974 2936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:55.158910 2933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:55.159677 2928 server_base.cc:1034] running on GCE node
I20260928 05:10:55.160272 2928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:55.161129 2928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:55.162408 2928 hybrid_clock.cc:648] HybridClock initialized: now 1790572255162359 us; error 52 us; skew 500 ppm
I20260928 05:10:55.162804 2928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:55 dist-test-slave-6718 krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:55.166251 2928 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:10:55.167804 2928 webserver.cc:460] Webserver started at http://127.0.97.1:45929/ using document root <none> and password file <none>
I20260928 05:10:55.168462 2928 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:55.168568 2928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:55.168838 2928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:55.172040 2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "5bbb53b0b2f74fb7a407b669464e0c85"
format_stamp: "Formatted at 2026-09-28 05:10:55 on dist-test-slave-6718"
server_key: "55fb48ae6ce88de104498421abec5ea8"
server_key_iv: "c4d8ff4771f23cf7b1a2d60f20eb9efc"
server_key_version: "encryptionkey@0"
I20260928 05:10:55.172797 2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5bbb53b0b2f74fb7a407b669464e0c85"
format_stamp: "Formatted at 2026-09-28 05:10:55 on dist-test-slave-6718"
server_key: "55fb48ae6ce88de104498421abec5ea8"
server_key_iv: "c4d8ff4771f23cf7b1a2d60f20eb9efc"
server_key_version: "encryptionkey@0"
I20260928 05:10:55.178113 2928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:10:55.181751 2943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:55.183044 2928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:10:55.183218 2928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "5bbb53b0b2f74fb7a407b669464e0c85"
format_stamp: "Formatted at 2026-09-28 05:10:55 on dist-test-slave-6718"
server_key: "55fb48ae6ce88de104498421abec5ea8"
server_key_iv: "c4d8ff4771f23cf7b1a2d60f20eb9efc"
server_key_version: "encryptionkey@0"
I20260928 05:10:55.183403 2928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:55.201399 2928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:55.205746 2928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:55.206043 2928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:55.207105 2928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:55.208426 2928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:55.208539 2928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:55.208704 2928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:55.208775 2928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:55.240386 2928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46697
I20260928 05:10:55.240437 3055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46697 every 8 connection(s)
I20260928 05:10:55.241665 2928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:10:55.247231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 2928
I20260928 05:10:55.247418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:10:55.248035 388 external_mini_cluster.cc:1428] Setting key 7fd1628446c2a7cb2e63ae0b81c67482
Sep 28 05:10:55 dist-test-slave-6718 krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:55.262321 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:55.244177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50193 (local address 127.0.97.62:43311)
0928 05:10:55.244575 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:55.244579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:55.245668 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 05:10:55.247501 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:55.247506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:55.247594 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:55.247711 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:55.249465 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.250516 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.252026 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.252758 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.252909 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:55.256417 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:55.256423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:55.256426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:55.256457 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:55.259168 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:55.260057 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:55.260064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:55.260072 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:55.260182 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:55.260603 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:55.260610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:55.260613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:55.260877 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:55.261034 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:55.261716 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 05:10:55.261921 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":200,"threads_started":1}
I20260928 05:10:55.263293 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:55.244786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43311 (local address 127.0.97.1:50193)
0928 05:10:55.245505 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:55.245550 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:55.247201 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:55.247939 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:55.247947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:55.248491 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:55.249282 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:55.249307 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.250674 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.250678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:55.251880 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:55.251890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.252957 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.252961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:55.253149 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:55.253827 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:55.253851 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:55.256159 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:55.259318 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:55.259327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:55.259346 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:55.259926 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:55.260296 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:55.260301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:55.260305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:55.260493 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:55.261009 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:55.261017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:55.261363 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 05:10:55.261645 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 05:10:55.262160 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:10:55.264909 3056 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43311
I20260928 05:10:55.265208 3056 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:55.265955 3056 heartbeater.cc:511] Master 127.0.97.62:43311 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:10:55.268164 2867 ts_manager.cc:194] Registered new tserver with Master: 5bbb53b0b2f74fb7a407b669464e0c85 (127.0.97.1:46697)
I20260928 05:10:55.269523 2867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:10:55.287984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43311
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:55.620556 3062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:55.620923 3062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:55.621078 3062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:55.628055 3062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:55.628178 3062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:55.628360 3062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:10:55.637899 3062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:55.639186 3062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:55.640545 3062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:55.653486 3067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:55.653448 3068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:55.653517 3070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:55.654204 3062 server_base.cc:1034] running on GCE node
I20260928 05:10:55.654898 3062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:55.655799 3062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:55.657018 3062 hybrid_clock.cc:648] HybridClock initialized: now 1790572255656969 us; error 43 us; skew 500 ppm
I20260928 05:10:55.657330 3062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:55 dist-test-slave-6718 krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:55.660429 3062 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:10:55.661752 3062 webserver.cc:460] Webserver started at http://127.0.97.2:34633/ using document root <none> and password file <none>
I20260928 05:10:55.662422 3062 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:55.662595 3062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:55.662897 3062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:55.665663 3062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "345e9203f1c040bd8335a64b33eb029e"
format_stamp: "Formatted at 2026-09-28 05:10:55 on dist-test-slave-6718"
server_key: "dfb2135d63633da91f3552a9fe388b56"
server_key_iv: "77bfb7e50ac380ea92b8ca54061f5281"
server_key_version: "encryptionkey@0"
I20260928 05:10:55.666474 3062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "345e9203f1c040bd8335a64b33eb029e"
format_stamp: "Formatted at 2026-09-28 05:10:55 on dist-test-slave-6718"
server_key: "dfb2135d63633da91f3552a9fe388b56"
server_key_iv: "77bfb7e50ac380ea92b8ca54061f5281"
server_key_version: "encryptionkey@0"
I20260928 05:10:55.671428 3062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:10:55.674692 3077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:55.675963 3062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:10:55.676143 3062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "345e9203f1c040bd8335a64b33eb029e"
format_stamp: "Formatted at 2026-09-28 05:10:55 on dist-test-slave-6718"
server_key: "dfb2135d63633da91f3552a9fe388b56"
server_key_iv: "77bfb7e50ac380ea92b8ca54061f5281"
server_key_version: "encryptionkey@0"
I20260928 05:10:55.676366 3062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:55.702471 3062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:55.706358 3062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:55.706734 3062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:55.707660 3062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:55.708905 3062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:55.709005 3062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:55.709159 3062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:55.709232 3062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:55.741506 3062 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42965
I20260928 05:10:55.741549 3189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42965 every 8 connection(s)
I20260928 05:10:55.742795 3062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:10:55.744583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3062
I20260928 05:10:55.744748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:10:55.745203 388 external_mini_cluster.cc:1428] Setting key f598397749491783351f7883d412a17c
Sep 28 05:10:55 dist-test-slave-6718 krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:10:55.763696 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:55.746615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42197 (local address 127.0.97.62:43311)
0928 05:10:55.746780 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:55.746785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:55.747002 (+ 217us) server_negotiation.cc:407] Connection header received
0928 05:10:55.748759 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:55.748764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:55.748848 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:55.748959 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:55.750438 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.751179 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.752657 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.753048 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.753181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:55.756917 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:55.756927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:55.756935 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:55.756988 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:55.760698 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:55.761434 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:55.761442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:55.761446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:55.761600 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:55.762007 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:55.762015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:55.762020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:55.762409 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:55.762583 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:55.763297 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 05:10:55.763464 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:10:55.764775 3192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:55.746059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43311 (local address 127.0.97.2:42197)
0928 05:10:55.746834 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:55.746884 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:55.748510 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:55.749209 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:55.749221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:55.749739 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:55.750272 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:55.750287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.751317 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.751323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:55.752508 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:55.752523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:55.753265 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:55.753273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:55.753508 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:55.754270 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:55.754292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:55.756686 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:55.760856 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:55.760863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:55.760876 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:55.761309 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:55.761715 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:55.761718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:55.761720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:55.761881 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:55.762583 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:55.762594 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:55.762962 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 05:10:55.763194 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 05:10:55.763592 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":315,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:10:55.766288 3190 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43311
I20260928 05:10:55.766659 3190 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:55.767374 3190 heartbeater.cc:511] Master 127.0.97.62:43311 requested a full tablet report, sending...
I20260928 05:10:55.768821 2867 ts_manager.cc:194] Registered new tserver with Master: 345e9203f1c040bd8335a64b33eb029e (127.0.97.2:42965)
I20260928 05:10:55.769622 2867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:10:55.789489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43311
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:56.102437 3195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:56.102789 3195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:56.102945 3195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:56.113045 3195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:56.113193 3195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:56.113387 3195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:10:56.123080 3195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:56.124352 3195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:56.125594 3195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:56.137951 3200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:56.137989 3201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:56.138161 3203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:56.138841 3195 server_base.cc:1034] running on GCE node
I20260928 05:10:56.139425 3195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:56.140300 3195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:56.141523 3195 hybrid_clock.cc:648] HybridClock initialized: now 1790572256141487 us; error 55 us; skew 500 ppm
I20260928 05:10:56.141821 3195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:56 dist-test-slave-6718 krb5kdc[2822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:56.144722 3195 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:10:56.146013 3195 webserver.cc:460] Webserver started at http://127.0.97.3:43543/ using document root <none> and password file <none>
I20260928 05:10:56.146736 3195 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:56.146875 3195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:56.147172 3195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:56.149864 3195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "9be1cd27a0224179b8b03b05a26d0e12"
format_stamp: "Formatted at 2026-09-28 05:10:56 on dist-test-slave-6718"
server_key: "38266a33e3713e6921cb1157e8f70ff4"
server_key_iv: "0cf89bbac7e01bf73dcea5f293572d86"
server_key_version: "encryptionkey@0"
I20260928 05:10:56.150667 3195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9be1cd27a0224179b8b03b05a26d0e12"
format_stamp: "Formatted at 2026-09-28 05:10:56 on dist-test-slave-6718"
server_key: "38266a33e3713e6921cb1157e8f70ff4"
server_key_iv: "0cf89bbac7e01bf73dcea5f293572d86"
server_key_version: "encryptionkey@0"
I20260928 05:10:56.155588 3195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:10:56.158910 3210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:56.160181 3195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:10:56.160348 3195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "9be1cd27a0224179b8b03b05a26d0e12"
format_stamp: "Formatted at 2026-09-28 05:10:56 on dist-test-slave-6718"
server_key: "38266a33e3713e6921cb1157e8f70ff4"
server_key_iv: "0cf89bbac7e01bf73dcea5f293572d86"
server_key_version: "encryptionkey@0"
I20260928 05:10:56.160550 3195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:56.202023 3195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:56.205919 3195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:56.206223 3195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:56.207170 3195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:10:56.208432 3195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:10:56.208534 3195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:56.208674 3195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:10:56.208743 3195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:56.241830 3195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36997
I20260928 05:10:56.241868 3322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36997 every 8 connection(s)
I20260928 05:10:56.243155 3195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:10:56.246182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3195
I20260928 05:10:56.246338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:10:56.246829 388 external_mini_cluster.cc:1428] Setting key 120c4019c95b14430be13b7dc2dd25de
Sep 28 05:10:56 dist-test-slave-6718 krb5kdc[2822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:56.262146 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:56.245598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57283 (local address 127.0.97.62:43311)
0928 05:10:56.245774 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:56.245779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:56.246991 (+ 1212us) server_negotiation.cc:407] Connection header received
0928 05:10:56.249066 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:56.249071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:56.249154 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:56.249256 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:56.250676 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:56.251383 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:56.252878 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:56.253269 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:56.253466 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:56.256817 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:56.256822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:56.256826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:56.256856 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:56.259244 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:56.259944 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:56.259954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:56.259956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:56.260066 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:56.260482 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:56.260487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:56.260490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:56.260769 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:56.260918 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:56.261594 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 05:10:56.261802 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:10:56.263146 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:56.246046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43311 (local address 127.0.97.3:57283)
0928 05:10:56.246818 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 05:10:56.246866 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:10:56.248810 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:10:56.249482 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:10:56.249490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:10:56.249958 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:10:56.250473 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:56.250490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:56.251533 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:56.251539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:56.252732 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:10:56.252748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:10:56.253533 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:10:56.253542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:10:56.253780 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:56.254428 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:10:56.254450 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:10:56.256600 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:10:56.259384 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:56.259394 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:56.259411 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:56.259815 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:56.260173 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:10:56.260179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:10:56.260183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:10:56.260361 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:10:56.260898 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:10:56.260905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:10:56.261225 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 05:10:56.261480 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 05:10:56.262010 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":338,"threads_started":1}
I20260928 05:10:56.264333 3323 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43311
I20260928 05:10:56.264791 3323 heartbeater.cc:464] Registering TS with master...
I20260928 05:10:56.265609 3323 heartbeater.cc:511] Master 127.0.97.62:43311 requested a full tablet report, sending...
I20260928 05:10:56.267038 2867 ts_manager.cc:194] Registered new tserver with Master: 9be1cd27a0224179b8b03b05a26d0e12 (127.0.97.3:36997)
I20260928 05:10:56.267834 2867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57283
I20260928 05:10:56.272593 3056 heartbeater.cc:503] Master 127.0.97.62:43311 was elected leader, sending a full tablet report...
I20260928 05:10:56.273993 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:10:56.302824 3058 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:10:56.300277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.97.62:43311)
0928 05:10:56.300498 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:56.300507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:56.300909 (+ 402us) server_negotiation.cc:407] Connection header received
0928 05:10:56.301119 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:56.301123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:56.301183 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:56.301289 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:56.302564 (+ 1275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35410: BlockingRecv error: recv got EOF from 127.0.0.1:35410 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:10:56.305840 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2928
W20260928 05:10:56.308450 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2928: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:46697: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:10:56.308633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 2928
I20260928 05:10:56.323735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3062
W20260928 05:10:56.326392 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3062: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42965: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:10:56.326568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3062
I20260928 05:10:56.341778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3195
W20260928 05:10:56.344429 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3195: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36997: server requires authentication, but client does not have Kerberos credentials available
I20260928 05:10:56.344594 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3195
I20260928 05:10:56.359234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 2835
I20260928 05:10:56.360461 2897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:56.462770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 2835
2026-09-28T05:10:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3526 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:56 dist-test-slave-6718 krb5kdc[3340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:56 dist-test-slave-6718 krb5kdc[3340](info): set up 2 sockets
Sep 28 05:10:56 dist-test-slave-6718 krb5kdc[3340](info): commencing operation
krb5kdc: starting...
W20260928 05:10:57.542745 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:10:57 dist-test-slave-6718 krb5kdc[3340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:10:57.727322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36245
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:58.039944 3353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:58.040279 3353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:58.040465 3353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:58.047793 3353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:58.047966 3353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:58.048067 3353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:58.048168 3353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:58.048249 3353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:58.058241 3353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:58.059579 3353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:58.060846 3353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:58.072171 3359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:58.072247 3361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:58.072202 3358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:58.072842 3353 server_base.cc:1034] running on GCE node
I20260928 05:10:58.073514 3353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:58.074813 3353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:58.076049 3353 hybrid_clock.cc:648] HybridClock initialized: now 1790572258076006 us; error 53 us; skew 500 ppm
I20260928 05:10:58.076378 3353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:58 dist-test-slave-6718 krb5kdc[3340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:58.083031 3353 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:58.084625 3353 webserver.cc:460] Webserver started at http://127.0.97.62:33147/ using document root <none> and password file <none>
I20260928 05:10:58.085343 3353 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:58.085489 3353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:58.085834 3353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:58.088884 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "15daaab60d4d4df69b3f4af848b8ac65"
format_stamp: "Formatted at 2026-09-28 05:10:58 on dist-test-slave-6718"
server_key: "0e141dd675df5d7463e228e5406eb0f8"
server_key_iv: "bd9c3439afe889d9964d95b288168534"
server_key_version: "encryptionkey@0"
I20260928 05:10:58.089672 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "15daaab60d4d4df69b3f4af848b8ac65"
format_stamp: "Formatted at 2026-09-28 05:10:58 on dist-test-slave-6718"
server_key: "0e141dd675df5d7463e228e5406eb0f8"
server_key_iv: "bd9c3439afe889d9964d95b288168534"
server_key_version: "encryptionkey@0"
I20260928 05:10:58.094803 3353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:10:58.098062 3368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:58.099400 3353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:10:58.099602 3353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "15daaab60d4d4df69b3f4af848b8ac65"
format_stamp: "Formatted at 2026-09-28 05:10:58 on dist-test-slave-6718"
server_key: "0e141dd675df5d7463e228e5406eb0f8"
server_key_iv: "bd9c3439afe889d9964d95b288168534"
server_key_version: "encryptionkey@0"
I20260928 05:10:58.099804 3353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:10:58.121160 3353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:10:58.124945 3353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:10:58.125223 3353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:10:58.142741 3353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36245
I20260928 05:10:58.142763 3419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36245 every 8 connection(s)
I20260928 05:10:58.144047 3353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:10:58.147575 3420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:10:58.154256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3353
I20260928 05:10:58.154493 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:10:58.155243 388 external_mini_cluster.cc:1428] Setting key 243e37fc5ff5775e49c802cf6a449ad2
I20260928 05:10:58.158463 3420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65: Bootstrap starting.
I20260928 05:10:58.161484 3420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65: Neither blocks nor log segments found. Creating new log.
I20260928 05:10:58.162514 3420 log.cc:824] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65: Log is configured to *not* fsync() on all Append() calls
I20260928 05:10:58.165593 3420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65: No bootstrap required, opened a new log
Sep 28 05:10:58 dist-test-slave-6718 krb5kdc[3340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:10:58.176376 3420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15daaab60d4d4df69b3f4af848b8ac65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36245 } }
I20260928 05:10:58.176832 3420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:10:58.176954 3420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15daaab60d4d4df69b3f4af848b8ac65, State: Initialized, Role: FOLLOWER
I20260928 05:10:58.177665 3420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15daaab60d4d4df69b3f4af848b8ac65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36245 } }
I20260928 05:10:58.177995 3420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:10:58.178139 3420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:10:58.178316 3420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:10:58.179709 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:58.157124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.0.97.62:36245)
0928 05:10:58.157852 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:58.157863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:58.157902 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:10:58.159633 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:58.159656 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:58.160071 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:58.160454 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:58.161881 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:58.163535 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:58.165353 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:58.166090 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:58.166243 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:58.170814 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:58.170824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:58.170849 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:10:58.170914 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:58.175010 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:58.175671 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:58.175679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:58.175689 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:58.175901 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:10:58.176253 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:10:58.176257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:10:58.176260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:10:58.176995 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:58.177275 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:58.177751 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 05:10:58.178239 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260928 05:10:58.181136 3420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15daaab60d4d4df69b3f4af848b8ac65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36245 } }
I20260928 05:10:58.181649 3420 leader_election.cc:302] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15daaab60d4d4df69b3f4af848b8ac65; no voters:
I20260928 05:10:58.182381 3420 leader_election.cc:288] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:10:58.182440 3425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:10:58.183745 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:10:58.183818 3425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [term 1 LEADER]: Becoming Leader. State: Replica: 15daaab60d4d4df69b3f4af848b8ac65, State: Running, Role: LEADER
I20260928 05:10:58.184312 3425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15daaab60d4d4df69b3f4af848b8ac65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36245 } }
I20260928 05:10:58.184573 3420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:10:58.187033 3426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15daaab60d4d4df69b3f4af848b8ac65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15daaab60d4d4df69b3f4af848b8ac65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36245 } } }
I20260928 05:10:58.186960 3427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15daaab60d4d4df69b3f4af848b8ac65. Latest consensus state: current_term: 1 leader_uuid: "15daaab60d4d4df69b3f4af848b8ac65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15daaab60d4d4df69b3f4af848b8ac65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36245 } } }
I20260928 05:10:58.187330 3426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:58.187330 3427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65 [sys.catalog]: This master's current role is: LEADER
I20260928 05:10:58.188145 3430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:10:58.197922 3430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:10:58.207825 3430 catalog_manager.cc:1347] Generated new cluster ID: 184e2518dbb84cdeadca0adf30e63723
I20260928 05:10:58.207952 3430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:10:58.221532 3430 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:10:58.223459 3430 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:10:58.239591 3430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15daaab60d4d4df69b3f4af848b8ac65: Generated new TSK 0
I20260928 05:10:58.240713 3430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:10:58.255434 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:10:58.245454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.0.97.62:36245)
0928 05:10:58.245731 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 05:10:58.245736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:10:58.245760 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:10:58.246122 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:10:58.246129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:10:58.246236 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:10:58.246311 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:10:58.247944 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:58.249149 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:58.251624 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:10:58.252442 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:10:58.252562 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:10:58.253911 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:10:58.253916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:10:58.253920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:10:58.254607 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:10:58.254670 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:10:58.254692 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:10:58.254783 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:10:58.254937 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 05:10:58.255028 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:10:58.262262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3353
I20260928 05:10:58.263564 3415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:10:58.355115 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3353
2026-09-28T05:10:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1886 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:10:58 dist-test-slave-6718 krb5kdc[3454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:10:58 dist-test-slave-6718 krb5kdc[3454](info): set up 2 sockets
Sep 28 05:10:58 dist-test-slave-6718 krb5kdc[3454](info): commencing operation
krb5kdc: starting...
W20260928 05:10:59.430925 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:10:59 dist-test-slave-6718 krb5kdc[3454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:10:59.617650 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43157
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:10:59.927817 3467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:10:59.928159 3467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:10:59.928326 3467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:10:59.935506 3467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:10:59.935654 3467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:10:59.935768 3467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:10:59.935863 3467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:10:59.935957 3467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:10:59.945765 3467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:10:59.947175 3467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:10:59.948448 3467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:10:59.959829 3472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:59.959785 3473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:10:59.959879 3475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:10:59.960732 3467 server_base.cc:1034] running on GCE node
I20260928 05:10:59.961477 3467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:10:59.962858 3467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:10:59.964090 3467 hybrid_clock.cc:648] HybridClock initialized: now 1790572259964055 us; error 51 us; skew 500 ppm
I20260928 05:10:59.964411 3467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:10:59 dist-test-slave-6718 krb5kdc[3454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:10:59.967371 3467 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:10:59.968809 3467 webserver.cc:460] Webserver started at http://127.0.97.62:44183/ using document root <none> and password file <none>
I20260928 05:10:59.969476 3467 fs_manager.cc:359] Metadata directory not provided
I20260928 05:10:59.969619 3467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:10:59.969918 3467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:10:59.972905 3467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "6889d24a82524c7f92754bc001995af4"
format_stamp: "Formatted at 2026-09-28 05:10:59 on dist-test-slave-6718"
server_key: "8c8d9a897263a66f843abbb5fa7351c0"
server_key_iv: "825def863b29153bdd711bc8691f9d87"
server_key_version: "encryptionkey@0"
I20260928 05:10:59.973659 3467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "6889d24a82524c7f92754bc001995af4"
format_stamp: "Formatted at 2026-09-28 05:10:59 on dist-test-slave-6718"
server_key: "8c8d9a897263a66f843abbb5fa7351c0"
server_key_iv: "825def863b29153bdd711bc8691f9d87"
server_key_version: "encryptionkey@0"
I20260928 05:10:59.978895 3467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:10:59.982190 3482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:10:59.983645 3467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:10:59.983834 3467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "6889d24a82524c7f92754bc001995af4"
format_stamp: "Formatted at 2026-09-28 05:10:59 on dist-test-slave-6718"
server_key: "8c8d9a897263a66f843abbb5fa7351c0"
server_key_iv: "825def863b29153bdd711bc8691f9d87"
server_key_version: "encryptionkey@0"
I20260928 05:10:59.984033 3467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:00.022704 3467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:00.026710 3467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:00.027011 3467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:00.045276 3467 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43157
I20260928 05:11:00.045280 3533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43157 every 8 connection(s)
I20260928 05:11:00.046665 3467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:00.050338 3534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:00.055630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3467
I20260928 05:11:00.055886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:00.056478 388 external_mini_cluster.cc:1428] Setting key a6a7b0a358498c45ae10919fd0597bea
I20260928 05:11:00.060838 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4: Bootstrap starting.
I20260928 05:11:00.064352 3534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:00.065570 3534 log.cc:824] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:00.068723 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4: No bootstrap required, opened a new log
Sep 28 05:11:00 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:00.079336 3534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6889d24a82524c7f92754bc001995af4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260928 05:11:00.079768 3534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:00.079875 3534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6889d24a82524c7f92754bc001995af4, State: Initialized, Role: FOLLOWER
I20260928 05:11:00.080090 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:00.058419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.0.97.62:43157)
0928 05:11:00.059269 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:00.059280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:00.059319 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:11:00.060891 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:00.060910 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:00.061321 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:00.061626 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:00.063335 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:00.064656 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:00.066440 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:00.067281 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:00.067417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:00.071848 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:00.071857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:00.071877 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:00.071928 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:00.075765 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:00.076391 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:00.076399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:00.076412 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:00.076618 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:00.076959 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:00.076965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:00.076973 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:00.077686 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:00.078006 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:00.078296 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 05:11:00.078744 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":393,"threads_started":1}
I20260928 05:11:00.080514 3534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6889d24a82524c7f92754bc001995af4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260928 05:11:00.080746 3534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:00.080901 3534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:00.081090 3534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:00.083367 3534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6889d24a82524c7f92754bc001995af4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260928 05:11:00.083828 3534 leader_election.cc:302] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6889d24a82524c7f92754bc001995af4; no voters:
I20260928 05:11:00.084378 3534 leader_election.cc:288] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:00.084527 3539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:00.085171 3539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [term 1 LEADER]: Becoming Leader. State: Replica: 6889d24a82524c7f92754bc001995af4, State: Running, Role: LEADER
I20260928 05:11:00.085632 3539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6889d24a82524c7f92754bc001995af4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260928 05:11:00.086275 3534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:00.088971 3540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6889d24a82524c7f92754bc001995af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6889d24a82524c7f92754bc001995af4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } } }
I20260928 05:11:00.088994 3541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6889d24a82524c7f92754bc001995af4. Latest consensus state: current_term: 1 leader_uuid: "6889d24a82524c7f92754bc001995af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6889d24a82524c7f92754bc001995af4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } } }
I20260928 05:11:00.089356 3540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:00.089416 3541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:00.090358 3544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:00.096580 3544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:00.104697 3544 catalog_manager.cc:1347] Generated new cluster ID: 73f9bd6d4a314bf280839e235a354328
I20260928 05:11:00.104807 3544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:00.123342 3544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:00.125161 3544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:00.135604 3544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6889d24a82524c7f92754bc001995af4: Generated new TSK 0
I20260928 05:11:00.136631 3544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:11:00.182024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:00.497897 3560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:00.498162 3560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:00.498309 3560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:00.505442 3560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:00.505555 3560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:00.505748 3560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:00.515223 3560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:00.516508 3560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:00.517788 3560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:00.530737 3565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:00.530781 3566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:00.530786 3568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:00.531611 3560 server_base.cc:1034] running on GCE node
I20260928 05:11:00.532264 3560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:00.533147 3560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:00.534354 3560 hybrid_clock.cc:648] HybridClock initialized: now 1790572260534314 us; error 29 us; skew 500 ppm
I20260928 05:11:00.534718 3560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:00 dist-test-slave-6718 krb5kdc[3454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:00.537705 3560 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:11:00.539120 3560 webserver.cc:460] Webserver started at http://127.0.97.1:42659/ using document root <none> and password file <none>
I20260928 05:11:00.539760 3560 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:00.539852 3560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:00.540084 3560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:00.542936 3560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "3868eef11ead43e6959ebc0190c1eb7c"
format_stamp: "Formatted at 2026-09-28 05:11:00 on dist-test-slave-6718"
server_key: "38ec6b19468e93c84e81fc077a4a8c97"
server_key_iv: "833959f425549ffcc9d2f1b2b848c3bf"
server_key_version: "encryptionkey@0"
I20260928 05:11:00.543642 3560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3868eef11ead43e6959ebc0190c1eb7c"
format_stamp: "Formatted at 2026-09-28 05:11:00 on dist-test-slave-6718"
server_key: "38ec6b19468e93c84e81fc077a4a8c97"
server_key_iv: "833959f425549ffcc9d2f1b2b848c3bf"
server_key_version: "encryptionkey@0"
I20260928 05:11:00.548357 3560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:11:00.551620 3575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:00.552911 3560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:11:00.553076 3560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "3868eef11ead43e6959ebc0190c1eb7c"
format_stamp: "Formatted at 2026-09-28 05:11:00 on dist-test-slave-6718"
server_key: "38ec6b19468e93c84e81fc077a4a8c97"
server_key_iv: "833959f425549ffcc9d2f1b2b848c3bf"
server_key_version: "encryptionkey@0"
I20260928 05:11:00.553273 3560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:00.583683 3560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:00.588399 3560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:00.588742 3560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:00.589768 3560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:00.591365 3560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:00.591471 3560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:00.591583 3560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:00.591627 3560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:00.623757 3560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44847
I20260928 05:11:00.623807 3687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44847 every 8 connection(s)
I20260928 05:11:00.625057 3560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:00.629964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3560
I20260928 05:11:00.630183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:00.630923 388 external_mini_cluster.cc:1428] Setting key 12c641336ca4b9e264abd62d5060a6bd
Sep 28 05:11:00 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:00.644137 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:00.627551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43397 (local address 127.0.97.62:43157)
0928 05:11:00.627946 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:00.627961 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:00.628884 (+ 923us) server_negotiation.cc:407] Connection header received
0928 05:11:00.630739 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:00.630747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:00.630836 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:00.630959 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:00.632421 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:00.633147 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:00.634736 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:00.635109 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:00.635241 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:00.638847 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:00.638852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:00.638855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:00.638884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:00.641223 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:00.641905 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:00.641911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:00.641916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:00.642026 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:00.642438 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:00.642443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:00.642446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:00.642763 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:00.642907 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:00.643563 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 05:11:00.643767 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":200,"threads_started":1}
I20260928 05:11:00.645087 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:00.628092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43157 (local address 127.0.97.1:43397)
0928 05:11:00.628721 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:00.628764 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:00.630420 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:00.631224 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:00.631231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:00.631693 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:00.632237 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:00.632256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:00.633299 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:00.633304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:00.634517 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:00.634579 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:00.635531 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:00.635536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:00.635731 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:00.636405 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:00.636429 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:00.638645 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:00.641357 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:00.641366 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:00.641382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:00.641782 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:00.642141 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:00.642146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:00.642149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:00.642324 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:00.642885 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:00.642892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:00.643219 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 05:11:00.643476 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 05:11:00.643997 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":254,"threads_started":1}
I20260928 05:11:00.646641 3688 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43157
I20260928 05:11:00.646936 3688 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:11:00.647677 3688 heartbeater.cc:511] Master 127.0.97.62:43157 requested a full tablet report, sending...
I20260928 05:11:00.649753 3499 ts_manager.cc:194] Registered new tserver with Master: 3868eef11ead43e6959ebc0190c1eb7c (127.0.97.1:44847)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:11:00.651319 3499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:11:00.672049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:00.988188 3694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:00.988515 3694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:00.988623 3694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:00.996165 3694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:00.996268 3694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:00.996382 3694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:01.005746 3694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:01.006997 3694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:01.008212 3694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:01.020785 3702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:01.020756 3699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:01.020848 3700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:01.021749 3694 server_base.cc:1034] running on GCE node
I20260928 05:11:01.022329 3694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:01.023293 3694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:01.024533 3694 hybrid_clock.cc:648] HybridClock initialized: now 1790572261024475 us; error 45 us; skew 500 ppm
I20260928 05:11:01.024848 3694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:01.028066 3694 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:11:01.029536 3694 webserver.cc:460] Webserver started at http://127.0.97.2:35953/ using document root <none> and password file <none>
I20260928 05:11:01.030223 3694 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:01.030366 3694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:01.030740 3694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:01.033631 3694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "bfa1c37d2f2343279f188c1f071b71e3"
format_stamp: "Formatted at 2026-09-28 05:11:01 on dist-test-slave-6718"
server_key: "b32beb4fda9f7eb9c101033eeb4a554e"
server_key_iv: "218a7a17209327b3d33dc7bb85b2ab70"
server_key_version: "encryptionkey@0"
I20260928 05:11:01.034413 3694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bfa1c37d2f2343279f188c1f071b71e3"
format_stamp: "Formatted at 2026-09-28 05:11:01 on dist-test-slave-6718"
server_key: "b32beb4fda9f7eb9c101033eeb4a554e"
server_key_iv: "218a7a17209327b3d33dc7bb85b2ab70"
server_key_version: "encryptionkey@0"
I20260928 05:11:01.039462 3694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:11:01.042778 3709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:01.044041 3694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:11:01.044204 3694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "bfa1c37d2f2343279f188c1f071b71e3"
format_stamp: "Formatted at 2026-09-28 05:11:01 on dist-test-slave-6718"
server_key: "b32beb4fda9f7eb9c101033eeb4a554e"
server_key_iv: "218a7a17209327b3d33dc7bb85b2ab70"
server_key_version: "encryptionkey@0"
I20260928 05:11:01.044428 3694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:01.057992 3694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:01.062252 3694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:01.062618 3694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:01.063536 3694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:01.064785 3694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:01.064882 3694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:01.065018 3694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:01.065090 3694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:01.097385 3694 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45109
I20260928 05:11:01.097409 3821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45109 every 8 connection(s)
I20260928 05:11:01.098628 3694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:01.108732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3694
I20260928 05:11:01.108928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:01.109504 388 external_mini_cluster.cc:1428] Setting key 9901c165f0b55493eb2b2914c1607f64
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:01.117877 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.100981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59399 (local address 127.0.97.62:43157)
0928 05:11:01.101177 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:01.101184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:01.102309 (+ 1125us) server_negotiation.cc:407] Connection header received
0928 05:11:01.104015 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:01.104019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:01.104094 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:01.104182 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:01.105497 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.106304 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.107763 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.108139 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.108232 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.112023 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:01.112028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:01.112032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:01.112059 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:01.114599 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.115462 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.115466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.115468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.115586 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.116069 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.116075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.116080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.116431 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:01.116578 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:01.117168 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 05:11:01.117415 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:11:01.118440 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.101514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43157 (local address 127.0.97.2:59399)
0928 05:11:01.102160 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:01.102202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:01.103782 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:01.104366 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:01.104373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:01.104823 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:01.105357 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:01.105373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.106420 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.106423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:01.107646 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:01.107657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.108308 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.108312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:01.108621 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.109318 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:01.109348 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:01.111821 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:01.114739 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:01.114750 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:01.114768 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:01.115352 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:01.115691 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:01.115699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:01.115705 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:01.115943 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:01.116549 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:01.116554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:01.116823 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 05:11:01.117026 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 05:11:01.117368 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":256,"threads_started":1}
I20260928 05:11:01.119800 3822 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43157
I20260928 05:11:01.120100 3822 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:01.120785 3822 heartbeater.cc:511] Master 127.0.97.62:43157 requested a full tablet report, sending...
I20260928 05:11:01.122117 3499 ts_manager.cc:194] Registered new tserver with Master: bfa1c37d2f2343279f188c1f071b71e3 (127.0.97.2:45109)
I20260928 05:11:01.123025 3499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59399
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:11:01.150192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:01.463176 3827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:01.463502 3827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:01.463667 3827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:01.470892 3827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:01.471040 3827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:01.471210 3827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:01.481009 3827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:01.482308 3827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:01.483625 3827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:01.496582 3832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:01.496672 3835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:01.496606 3833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:01.502883 3827 server_base.cc:1034] running on GCE node
I20260928 05:11:01.503454 3827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:01.504346 3827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:01.505560 3827 hybrid_clock.cc:648] HybridClock initialized: now 1790572261505502 us; error 72 us; skew 500 ppm
I20260928 05:11:01.505864 3827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:01.508819 3827 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:11:01.510080 3827 webserver.cc:460] Webserver started at http://127.0.97.3:40333/ using document root <none> and password file <none>
I20260928 05:11:01.510778 3827 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:01.510872 3827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:01.511104 3827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:01.513918 3827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "d550585fdb26454fba0cda9632505239"
format_stamp: "Formatted at 2026-09-28 05:11:01 on dist-test-slave-6718"
server_key: "51a588e0ccb8705cee8996632a29b511"
server_key_iv: "b49500b0a7cad8cf97048598ef574cd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:01.514739 3827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d550585fdb26454fba0cda9632505239"
format_stamp: "Formatted at 2026-09-28 05:11:01 on dist-test-slave-6718"
server_key: "51a588e0ccb8705cee8996632a29b511"
server_key_iv: "b49500b0a7cad8cf97048598ef574cd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:01.519680 3827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:11:01.522989 3842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:01.524428 3827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:11:01.524605 3827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "d550585fdb26454fba0cda9632505239"
format_stamp: "Formatted at 2026-09-28 05:11:01 on dist-test-slave-6718"
server_key: "51a588e0ccb8705cee8996632a29b511"
server_key_iv: "b49500b0a7cad8cf97048598ef574cd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:01.524873 3827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:01.538471 3827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:01.542259 3827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:01.542618 3827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:01.543555 3827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:01.544812 3827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:01.544893 3827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:01.545074 3827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:01.545128 3827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:01.577428 3827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36431
I20260928 05:11:01.577473 3954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36431 every 8 connection(s)
I20260928 05:11:01.578791 3827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:01.586377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3827
I20260928 05:11:01.586634 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:01.587137 388 external_mini_cluster.cc:1428] Setting key 7b8fa2cae6925a76c4a3bc4900039f3b
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:01.598464 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.581224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58043 (local address 127.0.97.62:43157)
0928 05:11:01.581388 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:01.581392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:01.582491 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 05:11:01.584287 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:01.584292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:01.584384 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:01.584491 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:01.585898 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.586750 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.588549 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.589140 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.589292 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.592931 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:01.592937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:01.592942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:01.592982 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:01.595481 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.596201 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.596208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.596211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.596336 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.596774 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.596780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.596784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.597061 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:01.597211 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:01.597899 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 05:11:01.598102 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:11:01.599433 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.581625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43157 (local address 127.0.97.3:58043)
0928 05:11:01.582306 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:01.582384 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:01.584027 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:01.584723 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:01.584732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:01.585204 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:01.585732 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:01.585748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.586891 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.586896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:01.588392 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:01.588429 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.589391 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.589401 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:01.589682 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.590422 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:01.590451 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:01.592715 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:01.595628 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:01.595638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:01.595654 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:01.596076 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:01.596455 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:01.596460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:01.596464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:01.596659 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:01.597189 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:01.597197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:01.597534 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 05:11:01.597800 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 05:11:01.598349 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
I20260928 05:11:01.600807 3955 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43157
I20260928 05:11:01.601164 3955 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:01.602104 3955 heartbeater.cc:511] Master 127.0.97.62:43157 requested a full tablet report, sending...
I20260928 05:11:01.603572 3499 ts_manager.cc:194] Registered new tserver with Master: d550585fdb26454fba0cda9632505239 (127.0.97.3:36431)
I20260928 05:11:01.604374 3499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58043
I20260928 05:11:01.614603 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:01.654850 3688 heartbeater.cc:503] Master 127.0.97.62:43157 was elected leader, sending a full tablet report...
I20260928 05:11:01.659889 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.646419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.97.62:43157)
0928 05:11:01.646724 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:01.646732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:01.647047 (+ 315us) server_negotiation.cc:407] Connection header received
0928 05:11:01.647181 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:01.647184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:01.647248 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:01.647336 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:01.648668 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.649484 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.650704 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.651072 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.651170 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.654754 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:01.654761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:01.654764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:01.654797 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:01.657526 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.658164 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.658168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.658171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.658286 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.658684 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.658688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.658690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.658981 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:01.659168 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:01.659397 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 05:11:01.659555 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 05:11:01.660382 3499 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51860
I20260928 05:11:01.661712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3560
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:01.677175 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.663356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.0.97.1:44847)
0928 05:11:01.663930 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:01.663937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:01.663960 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:11:01.664010 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:01.664016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:01.664227 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:01.664387 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:01.665500 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.666489 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.667737 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.668292 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.668448 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.672223 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:01.672234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:01.672248 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:01.672283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:01.675054 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.675628 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.675634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.675637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.675744 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.676037 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.676040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.676042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.676333 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:01.676545 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:01.676732 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 05:11:01.676903 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":283,"threads_started":1}
W20260928 05:11:01.678366 3683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41882
W20260928 05:11:01.679471 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3560: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:11:01.679663 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3560
I20260928 05:11:01.696099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3694
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:11:01.710933 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.696857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.0.97.2:45109)
0928 05:11:01.697324 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:01.697331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:01.697355 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:01.697408 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:01.697414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:01.697630 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:01.697798 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:01.699063 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.700047 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.701246 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.701779 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.701920 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.705883 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:01.705894 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:01.705908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:01.705939 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:01.708550 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.709131 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.709138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.709140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.709250 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.709542 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.709545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.709547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.709845 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:01.710065 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:01.710324 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 05:11:01.710572 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
W20260928 05:11:01.712170 3817 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42600
W20260928 05:11:01.713248 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3694: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:11:01.713420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3694
I20260928 05:11:01.728413 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3827
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572261, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:11:01.744104 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:01.729373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.0.97.3:36431)
0928 05:11:01.730061 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:01.730068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:01.730101 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:11:01.730159 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:01.730165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:01.730373 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:01.730566 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:01.731792 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.732795 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.733905 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:01.734421 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:01.734520 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:01.738701 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:01.738712 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:01.738726 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:01.738755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:01.741564 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.742139 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.742145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.742147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.742264 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:01.742816 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:01.742820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:01.742822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:01.743142 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:01.743364 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:01.743654 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 05:11:01.743836 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":372,"threads_started":1}
W20260928 05:11:01.745446 3950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42000
W20260928 05:11:01.746565 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3827: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:11:01.746737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3827
I20260928 05:11:01.761919 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3467
I20260928 05:11:01.763082 3529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:01.862215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3467
2026-09-28T05:11:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3512 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3972](info): set up 2 sockets
Sep 28 05:11:01 dist-test-slave-6718 krb5kdc[3972](info): commencing operation
krb5kdc: starting...
W20260928 05:11:02.942943 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:03 dist-test-slave-6718 krb5kdc[3972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:11:03.137944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39739
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:03.451107 3985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:03.451470 3985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:03.451638 3985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:03.458895 3985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:03.459041 3985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:03.459147 3985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:03.459249 3985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:03.459336 3985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:03.469224 3985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:03.470577 3985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:03.471869 3985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:03.483785 3993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:03.483805 3991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:03.484562 3990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:03.484658 3985 server_base.cc:1034] running on GCE node
I20260928 05:11:03.485603 3985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:03.487128 3985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:03.488349 3985 hybrid_clock.cc:648] HybridClock initialized: now 1790572263488280 us; error 57 us; skew 500 ppm
I20260928 05:11:03.488723 3985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:03 dist-test-slave-6718 krb5kdc[3972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:03.496078 3985 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:11:03.497685 3985 webserver.cc:460] Webserver started at http://127.0.97.62:44793/ using document root <none> and password file <none>
I20260928 05:11:03.498615 3985 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:03.498759 3985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:03.499105 3985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:03.502117 3985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "3d436ce5dfc141408a8652c93f35b037"
format_stamp: "Formatted at 2026-09-28 05:11:03 on dist-test-slave-6718"
server_key: "7a2a594fe41ed709489fc44b029bb003"
server_key_iv: "77f90db62ce5b644d2adc00e90dbd030"
server_key_version: "encryptionkey@0"
I20260928 05:11:03.502985 3985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d436ce5dfc141408a8652c93f35b037"
format_stamp: "Formatted at 2026-09-28 05:11:03 on dist-test-slave-6718"
server_key: "7a2a594fe41ed709489fc44b029bb003"
server_key_iv: "77f90db62ce5b644d2adc00e90dbd030"
server_key_version: "encryptionkey@0"
I20260928 05:11:03.508241 3985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:11:03.511631 4000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:03.512902 3985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:11:03.513098 3985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "3d436ce5dfc141408a8652c93f35b037"
format_stamp: "Formatted at 2026-09-28 05:11:03 on dist-test-slave-6718"
server_key: "7a2a594fe41ed709489fc44b029bb003"
server_key_iv: "77f90db62ce5b644d2adc00e90dbd030"
server_key_version: "encryptionkey@0"
I20260928 05:11:03.513286 3985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:03.528870 3985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:03.532927 3985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:03.533212 3985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:03.551044 3985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39739
I20260928 05:11:03.551079 4051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39739 every 8 connection(s)
I20260928 05:11:03.552347 3985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:03.554960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 3985
I20260928 05:11:03.555160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:03.555946 388 external_mini_cluster.cc:1428] Setting key 50007365ce34fd2362b5ee6128b19a29
I20260928 05:11:03.556567 4052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:03.570180 4052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037: Bootstrap starting.
Sep 28 05:11:03 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:03.573781 4052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:03.575198 4052 log.cc:824] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:03.578480 4052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037: No bootstrap required, opened a new log
I20260928 05:11:03.580658 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:03.558045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.0.97.62:39739)
0928 05:11:03.558842 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:03.558858 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:03.558917 (+ 59us) server_negotiation.cc:407] Connection header received
0928 05:11:03.561266 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:03.561296 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:03.561824 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:03.562259 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:03.563789 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:03.565626 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:03.566926 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:03.568018 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:03.568206 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:03.572044 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:03.572055 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:03.572079 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:03.572125 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:03.576072 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:03.576898 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:03.576907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:03.576914 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:03.577104 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:03.577499 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:03.577504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:03.577507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:03.578169 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:03.578436 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:03.578939 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 05:11:03.579368 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":316,"threads_started":1}
I20260928 05:11:03.587033 4052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d436ce5dfc141408a8652c93f35b037" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39739 } }
I20260928 05:11:03.587339 4052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:03.587412 4052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d436ce5dfc141408a8652c93f35b037, State: Initialized, Role: FOLLOWER
I20260928 05:11:03.587862 4052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d436ce5dfc141408a8652c93f35b037" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39739 } }
I20260928 05:11:03.588053 4052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:03.588145 4052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:03.588279 4052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:03.590222 4052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d436ce5dfc141408a8652c93f35b037" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39739 } }
I20260928 05:11:03.590814 4052 leader_election.cc:302] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d436ce5dfc141408a8652c93f35b037; no voters:
I20260928 05:11:03.591380 4052 leader_election.cc:288] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:03.591526 4057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:03.592195 4057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [term 1 LEADER]: Becoming Leader. State: Replica: 3d436ce5dfc141408a8652c93f35b037, State: Running, Role: LEADER
I20260928 05:11:03.592821 4057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d436ce5dfc141408a8652c93f35b037" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39739 } }
I20260928 05:11:03.593320 4052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:03.595500 4058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d436ce5dfc141408a8652c93f35b037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d436ce5dfc141408a8652c93f35b037" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39739 } } }
I20260928 05:11:03.595749 4059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d436ce5dfc141408a8652c93f35b037. Latest consensus state: current_term: 1 leader_uuid: "3d436ce5dfc141408a8652c93f35b037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d436ce5dfc141408a8652c93f35b037" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39739 } } }
I20260928 05:11:03.595952 4058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:03.596024 4059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:03.597059 4065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:03.602801 4065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:03.611652 4065 catalog_manager.cc:1347] Generated new cluster ID: 27fb1ebbfc48490bb3cdd63dd7a46ac6
I20260928 05:11:03.611786 4065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:03.618944 4065 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:03.620762 4065 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:03.640327 4065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d436ce5dfc141408a8652c93f35b037: Generated new TSK 0
I20260928 05:11:03.641203 4065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:11:03.698222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39739
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:04.019358 4078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:04.019644 4078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:04.019738 4078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:04.026964 4078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:04.027079 4078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:04.027199 4078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:04.037462 4078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:04.038738 4078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:04.039942 4078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:04.052515 4084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:04.052593 4086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:04.052563 4083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:04.053614 4078 server_base.cc:1034] running on GCE node
I20260928 05:11:04.054229 4078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:04.055369 4078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:04.056571 4078 hybrid_clock.cc:648] HybridClock initialized: now 1790572264056505 us; error 59 us; skew 500 ppm
I20260928 05:11:04.056897 4078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:04 dist-test-slave-6718 krb5kdc[3972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:04.060007 4078 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:11:04.061414 4078 webserver.cc:460] Webserver started at http://127.0.97.1:39025/ using document root <none> and password file <none>
I20260928 05:11:04.062041 4078 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:04.062136 4078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:04.062384 4078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:04.065312 4078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "355476e4f4a546cfb7523891ad1794c9"
format_stamp: "Formatted at 2026-09-28 05:11:04 on dist-test-slave-6718"
server_key: "4d772cd17e801e931918634ce62fdb23"
server_key_iv: "8937c1b34ad71f6f4597074bbe78d4c6"
server_key_version: "encryptionkey@0"
I20260928 05:11:04.066045 4078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "355476e4f4a546cfb7523891ad1794c9"
format_stamp: "Formatted at 2026-09-28 05:11:04 on dist-test-slave-6718"
server_key: "4d772cd17e801e931918634ce62fdb23"
server_key_iv: "8937c1b34ad71f6f4597074bbe78d4c6"
server_key_version: "encryptionkey@0"
I20260928 05:11:04.071127 4078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:11:04.074293 4093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:04.075687 4078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:04.075842 4078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "355476e4f4a546cfb7523891ad1794c9"
format_stamp: "Formatted at 2026-09-28 05:11:04 on dist-test-slave-6718"
server_key: "4d772cd17e801e931918634ce62fdb23"
server_key_iv: "8937c1b34ad71f6f4597074bbe78d4c6"
server_key_version: "encryptionkey@0"
I20260928 05:11:04.076046 4078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:04.094456 4078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:04.099372 4078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:04.099668 4078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:04.100701 4078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:04.102069 4078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:04.102138 4078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:04.102298 4078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:04.102353 4078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:04.138523 4078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37487
I20260928 05:11:04.138629 4205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37487 every 8 connection(s)
I20260928 05:11:04.139839 4078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:04.146745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4078
I20260928 05:11:04.146953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:04.147651 388 external_mini_cluster.cc:1428] Setting key 675d06fb54aa34b933324966cc05f109
Sep 28 05:11:04 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:04.159691 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:04.142376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55623 (local address 127.0.97.62:39739)
0928 05:11:04.142863 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:04.142870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:04.143782 (+ 912us) server_negotiation.cc:407] Connection header received
0928 05:11:04.145414 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:04.145418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:04.145485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:04.145579 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:04.147281 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.148063 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.149940 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.150630 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.150778 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:04.154232 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:04.154237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:04.154240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:04.154270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:04.156761 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:04.157442 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:04.157451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:04.157456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:04.157572 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:04.157990 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:04.157995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:04.157998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:04.158250 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:04.158400 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:04.159102 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 05:11:04.159315 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":256,"threads_started":1}
I20260928 05:11:04.160606 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:04.142917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39739 (local address 127.0.97.1:55623)
0928 05:11:04.143621 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:04.143658 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:04.145185 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:04.145770 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:04.145778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:04.146293 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:04.147113 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:04.147139 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.148231 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.148235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:04.149798 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:04.149811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.151093 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.151099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:04.151297 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:04.151962 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:04.151987 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:04.154023 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:04.156898 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:04.156907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:04.156923 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:04.157321 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:04.157679 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:04.157685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:04.157689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:04.157875 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:04.158373 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:04.158381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:04.158761 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 05:11:04.159026 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 05:11:04.159552 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":277,"threads_started":1}
I20260928 05:11:04.162119 4206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39739
I20260928 05:11:04.162459 4206 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:04.163409 4206 heartbeater.cc:511] Master 127.0.97.62:39739 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:11:04.165809 4017 ts_manager.cc:194] Registered new tserver with Master: 355476e4f4a546cfb7523891ad1794c9 (127.0.97.1:37487)
I20260928 05:11:04.167330 4017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:11:04.189868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39739
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:04.503085 4212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:04.503350 4212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:04.503522 4212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:04.510612 4212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:04.510743 4212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:04.510924 4212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:04.520547 4212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:04.522203 4212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:04.523670 4212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:04.536568 4217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:04.536537 4218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:04.536598 4220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:04.537425 4212 server_base.cc:1034] running on GCE node
I20260928 05:11:04.537995 4212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:04.538926 4212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:04.540151 4212 hybrid_clock.cc:648] HybridClock initialized: now 1790572264540104 us; error 56 us; skew 500 ppm
I20260928 05:11:04.540462 4212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:04 dist-test-slave-6718 krb5kdc[3972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:04.543617 4212 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:11:04.545084 4212 webserver.cc:460] Webserver started at http://127.0.97.2:39515/ using document root <none> and password file <none>
I20260928 05:11:04.545763 4212 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:04.545903 4212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:04.546197 4212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:04.549101 4212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "80851b39d4374ce0a2b8d84ce36d5280"
format_stamp: "Formatted at 2026-09-28 05:11:04 on dist-test-slave-6718"
server_key: "b2023414f59f60eeeb79d9442be98c37"
server_key_iv: "467a951648221df2846a1118b2cf3382"
server_key_version: "encryptionkey@0"
I20260928 05:11:04.549887 4212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "80851b39d4374ce0a2b8d84ce36d5280"
format_stamp: "Formatted at 2026-09-28 05:11:04 on dist-test-slave-6718"
server_key: "b2023414f59f60eeeb79d9442be98c37"
server_key_iv: "467a951648221df2846a1118b2cf3382"
server_key_version: "encryptionkey@0"
I20260928 05:11:04.555033 4212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:11:04.558449 4227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:04.559827 4212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:04.560006 4212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "80851b39d4374ce0a2b8d84ce36d5280"
format_stamp: "Formatted at 2026-09-28 05:11:04 on dist-test-slave-6718"
server_key: "b2023414f59f60eeeb79d9442be98c37"
server_key_iv: "467a951648221df2846a1118b2cf3382"
server_key_version: "encryptionkey@0"
I20260928 05:11:04.560207 4212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:04.581084 4212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:04.585196 4212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:04.585470 4212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:04.586407 4212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:04.587662 4212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:04.587769 4212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:04.587914 4212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:04.587988 4212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:04.619866 4212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41103
I20260928 05:11:04.619951 4339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41103 every 8 connection(s)
I20260928 05:11:04.621094 4212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:04.627104 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4212
I20260928 05:11:04.627285 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:04.627729 388 external_mini_cluster.cc:1428] Setting key 98281e3edfb54ac4c153f36e01c3a61d
Sep 28 05:11:04 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:04.641464 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:04.623665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43343 (local address 127.0.97.62:39739)
0928 05:11:04.623829 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:04.623833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:04.625076 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 05:11:04.626917 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:04.626923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:04.627013 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:04.627129 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:04.628778 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.629632 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.631195 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.631621 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.631754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:04.635226 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:04.635231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:04.635234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:04.635265 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:04.637761 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:04.638773 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:04.638776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:04.638778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:04.638908 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:04.639401 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:04.639403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:04.639405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:04.639647 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:04.639783 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:04.640724 (+ 941us) server_negotiation.cc:299] Negotiation successful
0928 05:11:04.641006 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:11:04.643985 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:04.624183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39739 (local address 127.0.97.2:43343)
0928 05:11:04.624915 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:04.624957 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:04.626636 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:04.627357 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:04.627365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:04.627950 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:04.628620 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:04.628639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.629768 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.629779 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:04.631057 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:04.631068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:04.631839 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:04.631846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:04.632069 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:04.632757 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:04.632782 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:04.634936 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:04.637918 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:04.637930 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:04.637953 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:04.638655 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:04.639020 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:04.639028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:04.639040 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:04.639292 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:04.639772 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:04.639784 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:04.640270 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 05:11:04.641704 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0928 05:11:04.642347 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:11:04.645604 4340 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39739
I20260928 05:11:04.645949 4340 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:04.646831 4340 heartbeater.cc:511] Master 127.0.97.62:39739 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:11:04.648458 4017 ts_manager.cc:194] Registered new tserver with Master: 80851b39d4374ce0a2b8d84ce36d5280 (127.0.97.2:41103)
I20260928 05:11:04.649652 4017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:11:04.671208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39739
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:04.984882 4345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:04.985199 4345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:04.985363 4345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:04.992445 4345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:04.992616 4345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:04.992789 4345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:05.002889 4345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:05.004158 4345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:05.005429 4345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:05.018649 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:05.018708 4353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:05.018649 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:05.019784 4345 server_base.cc:1034] running on GCE node
I20260928 05:11:05.020331 4345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:05.021210 4345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:05.022409 4345 hybrid_clock.cc:648] HybridClock initialized: now 1790572265022369 us; error 38 us; skew 500 ppm
I20260928 05:11:05.022771 4345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:05.025789 4345 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:11:05.027377 4345 webserver.cc:460] Webserver started at http://127.0.97.3:33257/ using document root <none> and password file <none>
I20260928 05:11:05.028009 4345 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:05.028105 4345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:05.028342 4345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:05.031194 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "020797329dfa4e8fb57c9dcf3adb14b6"
format_stamp: "Formatted at 2026-09-28 05:11:05 on dist-test-slave-6718"
server_key: "c36bcbd4ca7827576d6ededf406013c3"
server_key_iv: "23c06ef9e309ddf67328988de11a014d"
server_key_version: "encryptionkey@0"
I20260928 05:11:05.031914 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "020797329dfa4e8fb57c9dcf3adb14b6"
format_stamp: "Formatted at 2026-09-28 05:11:05 on dist-test-slave-6718"
server_key: "c36bcbd4ca7827576d6ededf406013c3"
server_key_iv: "23c06ef9e309ddf67328988de11a014d"
server_key_version: "encryptionkey@0"
I20260928 05:11:05.036918 4345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 05:11:05.040263 4360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:05.041543 4345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:11:05.041703 4345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "020797329dfa4e8fb57c9dcf3adb14b6"
format_stamp: "Formatted at 2026-09-28 05:11:05 on dist-test-slave-6718"
server_key: "c36bcbd4ca7827576d6ededf406013c3"
server_key_iv: "23c06ef9e309ddf67328988de11a014d"
server_key_version: "encryptionkey@0"
I20260928 05:11:05.041894 4345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:05.060732 4345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:05.064960 4345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:05.065245 4345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:05.066179 4345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:05.067976 4345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:05.068080 4345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:05.068228 4345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:05.068297 4345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:05.101912 4345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45543
I20260928 05:11:05.101967 4472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45543 every 8 connection(s)
I20260928 05:11:05.103282 4345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:05.108445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4345
I20260928 05:11:05.108580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:05.109033 388 external_mini_cluster.cc:1428] Setting key e941e1fee0520d7d4744f4f56a4a39e9
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:05.122488 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.105882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36969 (local address 127.0.97.62:39739)
0928 05:11:05.106100 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:05.106106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:05.107249 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 05:11:05.109147 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:05.109154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:05.109243 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:05.109369 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:05.110934 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.111834 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.113355 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.113746 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.113837 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.117252 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:05.117257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:05.117260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:05.117291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:05.119824 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.120498 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.120502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.120504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.120612 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.120985 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.120988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.120990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.121237 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:05.121341 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:05.121968 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 05:11:05.122159 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:11:05.123643 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.106325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39739 (local address 127.0.97.3:36969)
0928 05:11:05.107081 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:05.107118 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:05.108881 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:05.109568 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:05.109576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:05.110139 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:05.110797 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:05.110817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.112019 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.112023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:05.113235 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:05.113246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.113947 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.113952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:05.114138 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.114851 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:05.114876 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:05.117074 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:05.119959 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:05.119966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:05.119977 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:05.120360 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:05.120700 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:05.120704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:05.120706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:05.120881 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:05.121337 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:05.121343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:05.121640 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 05:11:05.121849 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 05:11:05.122205 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":341,"threads_started":1}
I20260928 05:11:05.125056 4473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39739
I20260928 05:11:05.125298 4473 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:05.125936 4473 heartbeater.cc:511] Master 127.0.97.62:39739 requested a full tablet report, sending...
I20260928 05:11:05.127295 4017 ts_manager.cc:194] Registered new tserver with Master: 020797329dfa4e8fb57c9dcf3adb14b6 (127.0.97.3:45543)
I20260928 05:11:05.128011 4017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36969
I20260928 05:11:05.136610 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 05:11:05.170243 4206 heartbeater.cc:503] Master 127.0.97.62:39739 was elected leader, sending a full tablet report...
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:05.184587 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.169469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.0.97.1:37487)
0928 05:11:05.170241 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:05.170250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:05.170287 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:11:05.170362 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:05.170370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:05.170679 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:05.170860 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:05.172198 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.173168 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.174383 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.175025 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.175166 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.178976 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:05.178988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:05.179003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:05.179036 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:05.181787 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.182421 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.182429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.182431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.182608 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.182971 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.182977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.182981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.183334 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:05.183640 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:05.183956 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 05:11:05.184227 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":314,"threads_started":1}
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:05.202369 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.188788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.0.97.62:39739)
0928 05:11:05.189026 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:05.189031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:05.189137 (+ 106us) server_negotiation.cc:407] Connection header received
0928 05:11:05.189263 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:05.189266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:05.189326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:05.189405 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:05.190666 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.191335 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.192720 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.193152 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.193244 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.197185 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:05.197190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:05.197193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:05.197224 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:05.199903 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.200628 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.200631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.200634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.200742 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.201128 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.201131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.201133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.201426 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:05.201608 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:05.201877 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 05:11:05.202030 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 05:11:05.202996 4017 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39626
I20260928 05:11:05.203975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4078
I20260928 05:11:05.217042 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.204889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.97.1:37487)
0928 05:11:05.205108 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:05.205114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:05.205309 (+ 195us) server_negotiation.cc:407] Connection header received
0928 05:11:05.205519 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:05.205529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:05.205591 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:05.205689 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:05.206966 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.207636 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.208856 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.209298 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.209430 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.211704 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:05.211709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:05.211713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:05.211745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:05.214457 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.215199 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.215206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.215210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.215332 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.215691 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.215698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.215701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.215949 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:05.216195 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:05.216497 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 05:11:05.216719 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:11:05.217633 4201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:05.302158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4078
I20260928 05:11:05.318152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4212
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:11:05.333766 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.319115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.97.2:41103)
0928 05:11:05.319610 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:05.319617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:05.319642 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:11:05.319699 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:05.319705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:05.319912 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:05.320097 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:05.321297 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.322302 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.323612 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.324166 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.324302 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.328265 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:05.328276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:05.328291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:05.328330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:05.331021 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.331646 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.331653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.331656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.331777 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.332142 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.332149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.332153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.332471 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:05.332762 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:05.333105 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 05:11:05.333356 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":226,"threads_started":1}
I20260928 05:11:05.335093 4335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:05.416955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4212
I20260928 05:11:05.435941 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4345
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[3972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:11:05.457304 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:05.437245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.0.97.3:45543)
0928 05:11:05.438072 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:05.438078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:05.438108 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:11:05.438172 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:05.438178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:05.438596 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:05.438788 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:05.440265 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.442120 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.444302 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:05.445069 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:05.445171 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:05.450600 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:05.450615 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:05.450637 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:05.450678 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:05.454412 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.455164 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.455173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.455177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.455312 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:05.455689 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:05.455694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:05.455697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:05.456071 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:05.456362 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:05.456683 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 05:11:05.456937 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":460,"threads_started":1}
I20260928 05:11:05.458998 4468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:05.548292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4345
I20260928 05:11:05.563859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 3985
I20260928 05:11:05.565001 4047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:05.665314 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 3985
2026-09-28T05:11:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3803 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[4492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[4492](info): set up 2 sockets
Sep 28 05:11:05 dist-test-slave-6718 krb5kdc[4492](info): commencing operation
krb5kdc: starting...
W20260928 05:11:06.755155 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:06 dist-test-slave-6718 krb5kdc[4492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:11:06.950227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40593
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:07.266772 4505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:07.267158 4505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:07.267339 4505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:07.275962 4505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:07.276085 4505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:07.276165 4505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:07.276264 4505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:07.276347 4505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:07.285884 4505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:07.287137 4505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:07.288371 4505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:07.299630 4511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:07.299631 4510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:07.299674 4513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:07.300539 4505 server_base.cc:1034] running on GCE node
I20260928 05:11:07.301275 4505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:07.302731 4505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:07.303959 4505 hybrid_clock.cc:648] HybridClock initialized: now 1790572267303919 us; error 50 us; skew 500 ppm
I20260928 05:11:07.304307 4505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:07 dist-test-slave-6718 krb5kdc[4492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:07.307355 4505 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:11:07.308434 4505 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:07.308517 4505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:07.308775 4505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:07.311585 4505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "2835e9fd53c5408295bbca3773af9270"
format_stamp: "Formatted at 2026-09-28 05:11:07 on dist-test-slave-6718"
server_key: "4786e09d9876ab23628ac67022408ad1"
server_key_iv: "af5bb656f43227e61dcbebe3ce4b730c"
server_key_version: "encryptionkey@0"
I20260928 05:11:07.312290 4505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "2835e9fd53c5408295bbca3773af9270"
format_stamp: "Formatted at 2026-09-28 05:11:07 on dist-test-slave-6718"
server_key: "4786e09d9876ab23628ac67022408ad1"
server_key_iv: "af5bb656f43227e61dcbebe3ce4b730c"
server_key_version: "encryptionkey@0"
I20260928 05:11:07.316956 4505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:11:07.320238 4519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:07.321442 4505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:11:07.321604 4505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "2835e9fd53c5408295bbca3773af9270"
format_stamp: "Formatted at 2026-09-28 05:11:07 on dist-test-slave-6718"
server_key: "4786e09d9876ab23628ac67022408ad1"
server_key_iv: "af5bb656f43227e61dcbebe3ce4b730c"
server_key_version: "encryptionkey@0"
I20260928 05:11:07.321821 4505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:07.349258 4505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:07.353217 4505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:07.353498 4505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:07.371191 4505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40593
I20260928 05:11:07.371212 4570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40593 every 8 connection(s)
I20260928 05:11:07.372049 4505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:07.375557 4571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:07.379117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4505
I20260928 05:11:07.379302 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:07.379906 388 external_mini_cluster.cc:1428] Setting key 6daccab7b25c810948a0ec5a086aa0fb
I20260928 05:11:07.386385 4571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270: Bootstrap starting.
I20260928 05:11:07.389282 4571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:07.390246 4571 log.cc:824] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:07.393505 4571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270: No bootstrap required, opened a new log
Sep 28 05:11:07 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:07.400956 4571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2835e9fd53c5408295bbca3773af9270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40593 } }
I20260928 05:11:07.401325 4571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:07.401459 4571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2835e9fd53c5408295bbca3773af9270, State: Initialized, Role: FOLLOWER
I20260928 05:11:07.402117 4571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2835e9fd53c5408295bbca3773af9270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40593 } }
I20260928 05:11:07.402401 4571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:07.402593 4571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:07.402791 4571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:07.403079 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:07.381800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.0.97.62:40593)
0928 05:11:07.382619 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:07.382631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:07.382670 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:11:07.384776 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:07.384796 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:07.385249 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:07.385698 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:07.387702 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:07.388962 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:07.390282 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:07.391360 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:07.391552 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:07.395987 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:07.395997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:07.396010 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:07.396058 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:07.399104 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:07.399756 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:07.399762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:07.399767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:07.399923 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:07.400313 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:07.400317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:07.400319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:07.400777 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:07.401086 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:07.401501 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 05:11:07.401909 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":370,"threads_started":1}
I20260928 05:11:07.405057 4571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2835e9fd53c5408295bbca3773af9270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40593 } }
I20260928 05:11:07.405597 4571 leader_election.cc:302] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2835e9fd53c5408295bbca3773af9270; no voters:
I20260928 05:11:07.406140 4571 leader_election.cc:288] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:07.406275 4576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:07.406980 4576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [term 1 LEADER]: Becoming Leader. State: Replica: 2835e9fd53c5408295bbca3773af9270, State: Running, Role: LEADER
I20260928 05:11:07.407495 4576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2835e9fd53c5408295bbca3773af9270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40593 } }
I20260928 05:11:07.408215 4571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:07.410173 4577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2835e9fd53c5408295bbca3773af9270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2835e9fd53c5408295bbca3773af9270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40593 } } }
I20260928 05:11:07.410467 4577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:07.411319 4583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:07.411181 4578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2835e9fd53c5408295bbca3773af9270. Latest consensus state: current_term: 1 leader_uuid: "2835e9fd53c5408295bbca3773af9270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2835e9fd53c5408295bbca3773af9270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40593 } } }
I20260928 05:11:07.411487 4578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:07.416661 4583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:07.427322 4583 catalog_manager.cc:1347] Generated new cluster ID: df14bdbe46ff43029670c1cfdd32c97b
I20260928 05:11:07.427443 4583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:07.467783 4583 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:07.469102 4583 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:07.478312 4583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270: Generated new TSK 0
I20260928 05:11:07.479182 4583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:11:07.530438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40593
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:07.869654 4597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:07.869916 4597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:07.870012 4597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:07.877059 4597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:07.877164 4597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:07.877275 4597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:07.886713 4597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:07.887878 4597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:07.889051 4597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:07.901319 4602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:07.901274 4603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:07.901355 4605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:07.902231 4597 server_base.cc:1034] running on GCE node
I20260928 05:11:07.902838 4597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:07.903838 4597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:07.905071 4597 hybrid_clock.cc:648] HybridClock initialized: now 1790572267905023 us; error 53 us; skew 500 ppm
I20260928 05:11:07.905385 4597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:07 dist-test-slave-6718 krb5kdc[4492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:07.908389 4597 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:11:07.909524 4597 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:07.909647 4597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:07.909972 4597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:07.912773 4597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "7b093b9e798d47898862e397c8c0032e"
format_stamp: "Formatted at 2026-09-28 05:11:07 on dist-test-slave-6718"
server_key: "c8316f6e6af22ec909f4bbe9f54fff68"
server_key_iv: "9eb8de1e8ef644078a2e84c93a206dc8"
server_key_version: "encryptionkey@0"
I20260928 05:11:07.913570 4597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7b093b9e798d47898862e397c8c0032e"
format_stamp: "Formatted at 2026-09-28 05:11:07 on dist-test-slave-6718"
server_key: "c8316f6e6af22ec909f4bbe9f54fff68"
server_key_iv: "9eb8de1e8ef644078a2e84c93a206dc8"
server_key_version: "encryptionkey@0"
I20260928 05:11:07.918484 4597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:11:07.921852 4611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:07.923195 4597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:07.923368 4597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "7b093b9e798d47898862e397c8c0032e"
format_stamp: "Formatted at 2026-09-28 05:11:07 on dist-test-slave-6718"
server_key: "c8316f6e6af22ec909f4bbe9f54fff68"
server_key_iv: "9eb8de1e8ef644078a2e84c93a206dc8"
server_key_version: "encryptionkey@0"
I20260928 05:11:07.923578 4597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:07.943259 4597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:07.947328 4597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:07.947640 4597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:07.948642 4597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:07.950127 4597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:07.950330 4597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:07.950516 4597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:07.950632 4597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 05:11:07.982481 4597 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39805
I20260928 05:11:07.982599 4723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39805 every 8 connection(s)
I20260928 05:11:07.983369 4597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:07.990630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4597
I20260928 05:11:07.990828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:07.991425 388 external_mini_cluster.cc:1428] Setting key e21b454440d804e323de91c3df65d542
Sep 28 05:11:07 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:08.003364 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:07.985733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58429 (local address 127.0.97.62:40593)
0928 05:11:07.986152 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:07.986159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:07.987168 (+ 1009us) server_negotiation.cc:407] Connection header received
0928 05:11:07.988879 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:07.988883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:07.988943 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:07.989041 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:07.990490 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:07.991380 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:07.993409 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:07.993811 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:07.993951 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:07.997514 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:07.997530 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:07.997539 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:07.997583 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:08.000213 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.001132 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.001136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.001138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.001242 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.001711 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.001714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.001716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.001957 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:08.002064 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:08.002969 (+ 905us) server_negotiation.cc:299] Negotiation successful
0928 05:11:08.003129 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":200,"threads_started":1}
I20260928 05:11:08.004949 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:07.986287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40593 (local address 127.0.97.1:58429)
0928 05:11:07.987003 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:07.987041 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:07.988634 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:07.989255 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:07.989263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:07.989746 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:07.990310 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:07.990326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:07.991528 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:07.991533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:07.993273 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:07.993287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:07.994034 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:07.994041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:07.994287 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:07.994975 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:07.994996 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:07.997296 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:08.000350 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:08.000362 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:08.000382 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:08.001019 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:08.001355 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:08.001362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:08.001368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:08.001609 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:08.002087 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:08.002098 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:08.002557 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 05:11:08.002883 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 05:11:08.003476 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":329,"threads_started":1}
I20260928 05:11:08.006614 4724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40593
I20260928 05:11:08.006918 4724 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:08.007565 4724 heartbeater.cc:511] Master 127.0.97.62:40593 requested a full tablet report, sending...
I20260928 05:11:08.009759 4536 ts_manager.cc:194] Registered new tserver with Master: 7b093b9e798d47898862e397c8c0032e (127.0.97.1:39805)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:11:08.011256 4536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:11:08.035398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40593
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:08.351728 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:08.351984 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:08.352146 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:08.359127 4730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:08.359228 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:08.359401 4730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:08.368795 4730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:08.370085 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:08.371765 4730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:08.385905 4735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:08.385878 4736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:08.386495 4738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:08.387276 4730 server_base.cc:1034] running on GCE node
I20260928 05:11:08.387898 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:08.388933 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:08.390158 4730 hybrid_clock.cc:648] HybridClock initialized: now 1790572268390098 us; error 52 us; skew 500 ppm
I20260928 05:11:08.390480 4730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:08 dist-test-slave-6718 krb5kdc[4492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:08.393831 4730 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:11:08.394977 4730 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:08.395085 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:08.395401 4730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:08.398121 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "2b64d946d0c9463889752d17f888776f"
format_stamp: "Formatted at 2026-09-28 05:11:08 on dist-test-slave-6718"
server_key: "3a8dcb90025f6cae03468dfd9d32eea0"
server_key_iv: "15d4ae66ad5562031e38f2653906e6a1"
server_key_version: "encryptionkey@0"
I20260928 05:11:08.398962 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2b64d946d0c9463889752d17f888776f"
format_stamp: "Formatted at 2026-09-28 05:11:08 on dist-test-slave-6718"
server_key: "3a8dcb90025f6cae03468dfd9d32eea0"
server_key_iv: "15d4ae66ad5562031e38f2653906e6a1"
server_key_version: "encryptionkey@0"
I20260928 05:11:08.403714 4730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:11:08.407164 4744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:08.408481 4730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:11:08.408643 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "2b64d946d0c9463889752d17f888776f"
format_stamp: "Formatted at 2026-09-28 05:11:08 on dist-test-slave-6718"
server_key: "3a8dcb90025f6cae03468dfd9d32eea0"
server_key_iv: "15d4ae66ad5562031e38f2653906e6a1"
server_key_version: "encryptionkey@0"
I20260928 05:11:08.408833 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:08.431438 4730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:08.435359 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:08.435624 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:08.436555 4730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:08.437808 4730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:08.437923 4730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:08.438045 4730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:08.438138 4730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:08.469364 4730 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38191
I20260928 05:11:08.469414 4856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38191 every 8 connection(s)
I20260928 05:11:08.470204 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:08.473382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4730
I20260928 05:11:08.473596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:08.474119 388 external_mini_cluster.cc:1428] Setting key 10a7e1ba28754684296ca7d7b718c48a
Sep 28 05:11:08 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:08.490782 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:08.472627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39507 (local address 127.0.97.62:40593)
0928 05:11:08.472805 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:08.472810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:08.474086 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 05:11:08.476190 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:08.476195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:08.476271 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:08.476379 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:08.477779 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.478985 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.480462 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.481218 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.481322 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:08.485138 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:08.485143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:08.485146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:08.485177 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:08.487640 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.488347 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.488358 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.488362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.488482 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.488886 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.488893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.488897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.489347 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:08.489519 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:08.490192 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 05:11:08.490423 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:11:08.491714 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:08.473083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40593 (local address 127.0.97.2:39507)
0928 05:11:08.473927 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:08.473970 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:08.475899 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:08.476578 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:08.476585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:08.477071 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:08.477605 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:08.477620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.479130 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.479134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:08.480322 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:08.480333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.481438 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.481447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:08.481758 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:08.482453 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:08.482495 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:08.484925 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:08.487777 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:08.487786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:08.487802 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:08.488221 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:08.488589 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:08.488596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:08.488602 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:08.488778 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:08.489506 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:08.489514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:08.489852 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 05:11:08.490114 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 05:11:08.490666 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":369,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:11:08.493105 4857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40593
I20260928 05:11:08.493427 4857 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:08.494033 4857 heartbeater.cc:511] Master 127.0.97.62:40593 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:11:08.495618 4536 ts_manager.cc:194] Registered new tserver with Master: 2b64d946d0c9463889752d17f888776f (127.0.97.2:38191)
I20260928 05:11:08.496876 4536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:11:08.517623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40593
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:08.830878 4862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:08.831197 4862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:08.831364 4862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:08.838449 4862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:08.838622 4862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:08.838797 4862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:08.848474 4862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:08.849862 4862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:08.851181 4862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:08.863416 4868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:08.863516 4870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:08.863575 4867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:08.864382 4862 server_base.cc:1034] running on GCE node
I20260928 05:11:08.864954 4862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:08.865818 4862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:08.867002 4862 hybrid_clock.cc:648] HybridClock initialized: now 1790572268866964 us; error 35 us; skew 500 ppm
I20260928 05:11:08.867323 4862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:08 dist-test-slave-6718 krb5kdc[4492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:08.874434 4862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:11:08.875626 4862 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:08.875713 4862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:08.875953 4862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:08.878794 4862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b"
format_stamp: "Formatted at 2026-09-28 05:11:08 on dist-test-slave-6718"
server_key: "67d3dd641af86aaeb6c0b4be6aac99de"
server_key_iv: "07b6a9b3f54f5e5b10ec857067bd2cf1"
server_key_version: "encryptionkey@0"
I20260928 05:11:08.879540 4862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b"
format_stamp: "Formatted at 2026-09-28 05:11:08 on dist-test-slave-6718"
server_key: "67d3dd641af86aaeb6c0b4be6aac99de"
server_key_iv: "07b6a9b3f54f5e5b10ec857067bd2cf1"
server_key_version: "encryptionkey@0"
I20260928 05:11:08.884348 4862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:11:08.887501 4876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:08.888768 4862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:08.888936 4862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b"
format_stamp: "Formatted at 2026-09-28 05:11:08 on dist-test-slave-6718"
server_key: "67d3dd641af86aaeb6c0b4be6aac99de"
server_key_iv: "07b6a9b3f54f5e5b10ec857067bd2cf1"
server_key_version: "encryptionkey@0"
I20260928 05:11:08.889151 4862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:08.905649 4862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:08.909613 4862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:08.909873 4862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:08.910808 4862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:08.912118 4862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:08.912226 4862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:08.912356 4862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:08.912448 4862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:08.943431 4862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40153
I20260928 05:11:08.943478 4988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40153 every 8 connection(s)
I20260928 05:11:08.944275 4862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:08.944859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 4862
I20260928 05:11:08.945066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:08.945603 388 external_mini_cluster.cc:1428] Setting key 4df9f74e30d240849cea9e944086b3f4
Sep 28 05:11:08 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:08.964048 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:08.946863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52729 (local address 127.0.97.62:40593)
0928 05:11:08.947062 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:08.947067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:08.948453 (+ 1386us) server_negotiation.cc:407] Connection header received
0928 05:11:08.950994 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:08.950998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:08.951060 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:08.951148 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:08.952538 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.953280 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.954786 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.955155 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.955246 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:08.958615 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:08.958620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:08.958623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:08.958651 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:08.961586 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.962204 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.962207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.962209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.962313 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.962724 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.962728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.962730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.962972 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:08.963084 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:08.963679 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 05:11:08.963825 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:11:08.964932 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:08.947494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40593 (local address 127.0.97.3:52729)
0928 05:11:08.948298 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:08.948342 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:08.950748 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:08.951347 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:08.951355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:08.951842 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:08.952393 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:08.952413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.953402 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.953406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:08.954667 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:08.954677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.955330 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.955334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:08.955507 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:08.956170 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:08.956191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:08.958381 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:08.961698 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:08.961703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:08.961715 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:08.962111 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:08.962400 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:08.962403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:08.962405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:08.962634 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:08.963075 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:08.963090 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:08.963362 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 05:11:08.963576 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 05:11:08.963925 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":339,"threads_started":1}
I20260928 05:11:08.966126 4989 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40593
I20260928 05:11:08.966418 4989 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:08.967059 4989 heartbeater.cc:511] Master 127.0.97.62:40593 requested a full tablet report, sending...
I20260928 05:11:08.968323 4536 ts_manager.cc:194] Registered new tserver with Master: 87d2a7eb08cf4d15b7d09070bf8ca26b (127.0.97.3:40153)
I20260928 05:11:08.969031 4536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52729
I20260928 05:11:08.972769 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:11:08.988842 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:08.976880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.0.97.62:40593)
0928 05:11:08.977028 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:08.977033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:08.977519 (+ 486us) server_negotiation.cc:407] Connection header received
0928 05:11:08.977643 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:08.977646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:08.977696 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:08.977771 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:08.979181 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.979828 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.980937 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:08.981313 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:08.981402 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:08.984112 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:08.984117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:08.984120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:08.984146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:08.986681 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.987389 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.987392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.987394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.987494 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:08.987785 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:08.987788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:08.987790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:08.988057 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:08.988231 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:08.988483 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 05:11:08.988641 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:11:08.992630 4536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:08.999271 4536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:09.014241 4724 heartbeater.cc:503] Master 127.0.97.62:40593 was elected leader, sending a full tablet report...
I20260928 05:11:09.020473 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.010331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38191 (local address 127.0.0.1:44562)
0928 05:11:09.011389 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:09.011418 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:09.011547 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:09.012541 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:09.012547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:09.012572 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:09.012979 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.012998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.015146 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.015155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.017522 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.017534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.019432 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.019440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.019671 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.019732 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 05:11:09.019838 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 05:11:09.020025 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":729,"threads_started":1}
I20260928 05:11:09.021516 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.010559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.0.97.2:38191)
0928 05:11:09.012019 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.012029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.012054 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:11:09.012170 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.012175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.012346 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.012546 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:09.013436 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.014974 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.017742 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.019198 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.020738 (+ 1540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.020941 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.021010 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.021208 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":1188,"threads_started":1}
I20260928 05:11:09.023455 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.013515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.0.97.1:39805)
0928 05:11:09.015757 (+ 2242us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.015764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.015790 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:11:09.015892 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.015900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.016115 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.016297 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:09.017200 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.019017 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.021122 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.022343 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.022468 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.022709 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.023005 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.023158 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":1920,"threads_started":1}
I20260928 05:11:09.023507 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.013821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39805 (local address 127.0.0.1:56702)
0928 05:11:09.015105 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:09.015122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:09.015219 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:09.016498 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:09.016503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:09.016527 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:09.016965 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.017009 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.019159 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.019167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.020972 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.020983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.022516 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.022522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.022815 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.022838 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:11:09.022954 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 05:11:09.023081 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":1,"thread_start_us":1003,"threads_started":1}
I20260928 05:11:09.026490 4792 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4c72573b0940b2ab3caee841a2384b (DEFAULT_TABLE table=test-table [id=07ad5f45f1b94926bbf9d6d49e47afb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:09.027324 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.011937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40153 (local address 127.0.0.1:34538)
0928 05:11:09.013166 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:09.013183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:09.013273 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:09.020174 (+ 6901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:09.020185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:09.020210 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:09.020571 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.020585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.022579 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.022584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.024732 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.024746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.026517 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.026523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.026737 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.026770 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:11:09.026884 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 05:11:09.027018 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":964,"threads_started":1}
I20260928 05:11:09.027310 4659 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4c72573b0940b2ab3caee841a2384b (DEFAULT_TABLE table=test-table [id=07ad5f45f1b94926bbf9d6d49e47afb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:09.028097 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4c72573b0940b2ab3caee841a2384b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:09.028251 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.012347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.0.97.3:40153)
0928 05:11:09.019625 (+ 7278us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.019633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.019663 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:11:09.019756 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.019764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.019977 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.020775 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:09.021009 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.022403 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.024910 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.026288 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.026412 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.026657 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.027754 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.027912 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7099,"thread_start_us":6940,"threads_started":1}
I20260928 05:11:09.028650 4659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4c72573b0940b2ab3caee841a2384b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:09.031682 4924 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4c72573b0940b2ab3caee841a2384b (DEFAULT_TABLE table=test-table [id=07ad5f45f1b94926bbf9d6d49e47afb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:09.032996 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4c72573b0940b2ab3caee841a2384b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:09.038808 5007 tablet_bootstrap.cc:492] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Bootstrap starting.
I20260928 05:11:09.041900 5007 tablet_bootstrap.cc:654] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:09.042140 5008 tablet_bootstrap.cc:492] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Bootstrap starting.
I20260928 05:11:09.043218 5007 log.cc:824] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:09.045766 5008 tablet_bootstrap.cc:654] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:09.046182 5007 tablet_bootstrap.cc:492] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: No bootstrap required, opened a new log
I20260928 05:11:09.046584 5007 ts_tablet_manager.cc:1397] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 05:11:09.046599 5010 tablet_bootstrap.cc:492] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Bootstrap starting.
I20260928 05:11:09.047035 5008 log.cc:824] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:09.049738 5010 tablet_bootstrap.cc:654] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:09.049996 5008 tablet_bootstrap.cc:492] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: No bootstrap required, opened a new log
I20260928 05:11:09.050263 5008 ts_tablet_manager.cc:1397] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 05:11:09.050928 5010 log.cc:824] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:09.054383 5007 raft_consensus.cc:348] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.054941 5007 raft_consensus.cc:374] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:09.055056 5007 raft_consensus.cc:729] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b093b9e798d47898862e397c8c0032e, State: Initialized, Role: FOLLOWER
I20260928 05:11:09.055699 5007 consensus_queue.cc:260] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.057143 5010 tablet_bootstrap.cc:492] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: No bootstrap required, opened a new log
I20260928 05:11:09.057443 5007 ts_tablet_manager.cc:1428] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Time spent starting tablet: real 0.011s user 0.000s sys 0.009s
I20260928 05:11:09.057443 5010 ts_tablet_manager.cc:1397] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260928 05:11:09.061370 5008 raft_consensus.cc:348] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.061815 5008 raft_consensus.cc:374] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:09.061932 5008 raft_consensus.cc:729] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b64d946d0c9463889752d17f888776f, State: Initialized, Role: FOLLOWER
I20260928 05:11:09.062567 5008 consensus_queue.cc:260] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.063730 4857 heartbeater.cc:503] Master 127.0.97.62:40593 was elected leader, sending a full tablet report...
I20260928 05:11:09.064213 5008 ts_tablet_manager.cc:1428] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260928 05:11:09.066216 5010 raft_consensus.cc:348] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.066643 5010 raft_consensus.cc:374] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:09.066751 5010 raft_consensus.cc:729] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d2a7eb08cf4d15b7d09070bf8ca26b, State: Initialized, Role: FOLLOWER
I20260928 05:11:09.067320 5010 consensus_queue.cc:260] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.068289 4989 heartbeater.cc:503] Master 127.0.97.62:40593 was elected leader, sending a full tablet report...
I20260928 05:11:09.068734 5010 ts_tablet_manager.cc:1428] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20260928 05:11:09.195952 4990 tablet.cc:2369] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:11:09.222165 4858 tablet.cc:2369] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:11:09.235523 4725 tablet.cc:2369] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:11:09.324987 5015 raft_consensus.cc:482] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:09.325258 5015 raft_consensus.cc:504] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.326992 5015 leader_election.cc:288] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b64d946d0c9463889752d17f888776f (127.0.97.2:38191), 7b093b9e798d47898862e397c8c0032e (127.0.97.1:39805)
I20260928 05:11:09.333256 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.327424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38705 (local address 127.0.97.2:38191)
0928 05:11:09.327636 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.327642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.327663 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:11:09.327810 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.327818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.327918 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.328019 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:09.328534 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.329496 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.331368 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.332394 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.332495 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.332561 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.332905 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.332991 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:11:09.333336 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.327265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38191 (local address 127.0.97.3:38705)
0928 05:11:09.327468 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:09.327490 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:09.327653 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:09.328165 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:09.328170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:09.328198 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:09.328375 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.328386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.329637 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.329642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.331214 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.331225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.332547 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.332553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.332744 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.332779 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:11:09.332878 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 05:11:09.333012 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:11:09.333859 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.327896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55635 (local address 127.0.97.1:39805)
0928 05:11:09.328105 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.328110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.328363 (+ 253us) server_negotiation.cc:407] Connection header received
0928 05:11:09.328507 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.328514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.328582 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.328647 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:09.329192 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.330103 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.332061 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.333003 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.333125 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.333222 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.333548 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.333669 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:11:09.334004 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.327649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39805 (local address 127.0.97.3:55635)
0928 05:11:09.328223 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:09.328236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:09.328395 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:09.328686 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:09.328690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:09.328714 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:09.329048 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.329058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.330249 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.330252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.331909 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:09.331922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.333198 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.333207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:09.333391 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.333428 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:11:09.333523 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 05:11:09.333663 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":280,"threads_started":1}
I20260928 05:11:09.334357 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c72573b0940b2ab3caee841a2384b" candidate_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b64d946d0c9463889752d17f888776f" is_pre_election: true
I20260928 05:11:09.334950 4812 raft_consensus.cc:2454] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d2a7eb08cf4d15b7d09070bf8ca26b in term 0.
I20260928 05:11:09.335013 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c72573b0940b2ab3caee841a2384b" candidate_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b093b9e798d47898862e397c8c0032e" is_pre_election: true
I20260928 05:11:09.335472 4679 raft_consensus.cc:2454] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d2a7eb08cf4d15b7d09070bf8ca26b in term 0.
I20260928 05:11:09.335556 4880 leader_election.cc:302] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b64d946d0c9463889752d17f888776f, 87d2a7eb08cf4d15b7d09070bf8ca26b; no voters:
I20260928 05:11:09.335956 5015 raft_consensus.cc:2781] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:09.336108 5015 raft_consensus.cc:482] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:09.336218 5015 raft_consensus.cc:3037] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:09.338285 5015 raft_consensus.cc:504] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.338908 5015 leader_election.cc:288] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [CANDIDATE]: Term 1 election: Requested vote from peers 2b64d946d0c9463889752d17f888776f (127.0.97.2:38191), 7b093b9e798d47898862e397c8c0032e (127.0.97.1:39805)
I20260928 05:11:09.339290 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c72573b0940b2ab3caee841a2384b" candidate_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b64d946d0c9463889752d17f888776f"
I20260928 05:11:09.339351 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4c72573b0940b2ab3caee841a2384b" candidate_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b093b9e798d47898862e397c8c0032e"
I20260928 05:11:09.339453 4812 raft_consensus.cc:3037] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:09.339556 4679 raft_consensus.cc:3037] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:09.341511 4812 raft_consensus.cc:2454] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d2a7eb08cf4d15b7d09070bf8ca26b in term 1.
I20260928 05:11:09.341566 4679 raft_consensus.cc:2454] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d2a7eb08cf4d15b7d09070bf8ca26b in term 1.
I20260928 05:11:09.341856 4880 leader_election.cc:302] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b64d946d0c9463889752d17f888776f, 87d2a7eb08cf4d15b7d09070bf8ca26b; no voters:
I20260928 05:11:09.342115 5015 raft_consensus.cc:2781] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:09.342598 5015 raft_consensus.cc:686] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 1 LEADER]: Becoming Leader. State: Replica: 87d2a7eb08cf4d15b7d09070bf8ca26b, State: Running, Role: LEADER
I20260928 05:11:09.343132 5015 consensus_queue.cc:237] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } }
I20260928 05:11:09.347193 4536 catalog_manager.cc:5495] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d2a7eb08cf4d15b7d09070bf8ca26b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d2a7eb08cf4d15b7d09070bf8ca26b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 } health_report { overall_health: UNKNOWN } } }
I20260928 05:11:09.395433 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.390165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.0.97.3:40153)
0928 05:11:09.390345 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.390352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.390371 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:11:09.390502 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.390506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.390653 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.390801 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:09.391181 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.391877 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.393305 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.393738 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.393886 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.394335 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:09.394889 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:09.395057 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.395113 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.395184 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:11:09.402957 4812 raft_consensus.cc:1261] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 1 FOLLOWER]: Refusing update from remote peer 87d2a7eb08cf4d15b7d09070bf8ca26b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:09.402947 4679 raft_consensus.cc:1261] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 1 FOLLOWER]: Refusing update from remote peer 87d2a7eb08cf4d15b7d09070bf8ca26b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:09.403872 5017 consensus_queue.cc:1035] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b64d946d0c9463889752d17f888776f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:09.404214 5015 consensus_queue.cc:1035] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b093b9e798d47898862e397c8c0032e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:09.416049 5020 mvcc.cc:205] Tried to move back new op lower bound from 7334184015464792064 to 7334184015233085440. Current Snapshot: MvccSnapshot[applied={T|T < 7334184015464792064}]
I20260928 05:11:09.416380 5021 mvcc.cc:205] Tried to move back new op lower bound from 7334184015464792064 to 7334184015233085440. Current Snapshot: MvccSnapshot[applied={T|T < 7334184015464792064}]
I20260928 05:11:09.428015 4535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:09.428277 4535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:09.431396 4535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2835e9fd53c5408295bbca3773af9270: Sending DeleteTablet for 3 replicas of tablet 0f4c72573b0940b2ab3caee841a2384b
I20260928 05:11:09.432343 4792 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4c72573b0940b2ab3caee841a2384b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44562
I20260928 05:11:09.432502 4659 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4c72573b0940b2ab3caee841a2384b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56702
I20260928 05:11:09.432507 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f4c72573b0940b2ab3caee841a2384b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34538
I20260928 05:11:09.434496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4597
I20260928 05:11:09.443609 5030 tablet_replica.cc:330] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: stopping tablet replica
I20260928 05:11:09.444842 5030 raft_consensus.cc:2229] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:09.446163 5030 raft_consensus.cc:2258] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:09.447033 5032 tablet_replica.cc:330] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: stopping tablet replica
I20260928 05:11:09.448159 5032 raft_consensus.cc:2229] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:09.449002 5031 tablet_replica.cc:330] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: stopping tablet replica
I20260928 05:11:09.449527 5031 raft_consensus.cc:2229] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:09.450017 5031 raft_consensus.cc:2258] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:09.450409 5032 raft_consensus.cc:2258] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:09.451583 5030 ts_tablet_manager.cc:1905] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:09.452193 5031 ts_tablet_manager.cc:1905] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:11:09 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:09.453590 5032 ts_tablet_manager.cc:1905] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:09.459851 5031 ts_tablet_manager.cc:1918] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:09.460106 5031 log.cc:1196] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/0f4c72573b0940b2ab3caee841a2384b
I20260928 05:11:09.460724 5031 ts_tablet_manager.cc:1939] T 0f4c72573b0940b2ab3caee841a2384b P 7b093b9e798d47898862e397c8c0032e: Deleting consensus metadata
I20260928 05:11:09.461766 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.436017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.97.1:39805)
0928 05:11:09.436180 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.436187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.441023 (+ 4836us) server_negotiation.cc:407] Connection header received
0928 05:11:09.441228 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.441233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.441329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.441430 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:09.444903 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.446092 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.447963 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.448766 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.448872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.454012 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:09.454028 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:09.454044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:09.454084 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:09.458154 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:09.459110 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:09.459118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:09.459122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:09.459280 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:09.459753 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:09.459758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:09.459762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:09.460222 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:09.460552 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.461116 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.461349 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:11:09.462005 5030 ts_tablet_manager.cc:1918] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:09.462664 4523 catalog_manager.cc:4841] TS 7b093b9e798d47898862e397c8c0032e (127.0.97.1:39805): tablet 0f4c72573b0940b2ab3caee841a2384b (table test-table [id=07ad5f45f1b94926bbf9d6d49e47afb0]) successfully deleted
I20260928 05:11:09.464591 4719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:09.475057 5032 ts_tablet_manager.cc:1918] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:09.475245 5032 log.cc:1196] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/0f4c72573b0940b2ab3caee841a2384b
I20260928 05:11:09.475796 5032 ts_tablet_manager.cc:1939] T 0f4c72573b0940b2ab3caee841a2384b P 87d2a7eb08cf4d15b7d09070bf8ca26b: Deleting consensus metadata
I20260928 05:11:09.477272 5030 log.cc:1196] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/0f4c72573b0940b2ab3caee841a2384b
I20260928 05:11:09.477352 4523 catalog_manager.cc:4841] TS 87d2a7eb08cf4d15b7d09070bf8ca26b (127.0.97.3:40153): tablet 0f4c72573b0940b2ab3caee841a2384b (table test-table [id=07ad5f45f1b94926bbf9d6d49e47afb0]) successfully deleted
I20260928 05:11:09.477855 5030 ts_tablet_manager.cc:1939] T 0f4c72573b0940b2ab3caee841a2384b P 2b64d946d0c9463889752d17f888776f: Deleting consensus metadata
I20260928 05:11:09.479439 4523 catalog_manager.cc:4841] TS 2b64d946d0c9463889752d17f888776f (127.0.97.2:38191): tablet 0f4c72573b0940b2ab3caee841a2384b (table test-table [id=07ad5f45f1b94926bbf9d6d49e47afb0]) successfully deleted
I20260928 05:11:09.600147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4597
I20260928 05:11:09.617942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4730
Sep 28 05:11:09 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:11:09.633909 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.618767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.0.97.2:38191)
0928 05:11:09.618952 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.618957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.619055 (+ 98us) server_negotiation.cc:407] Connection header received
0928 05:11:09.619225 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.619229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.619301 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.619381 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:09.620909 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.621625 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.623344 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.623729 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.623819 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.628587 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:09.628599 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:09.628615 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:09.628644 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:09.631365 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:09.632090 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:09.632096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:09.632098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:09.632232 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:09.632610 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:09.632613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:09.632615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:09.632931 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:09.633125 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.633455 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.633611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:11:09.634807 4852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:09.721823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4730
I20260928 05:11:09.738234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4862
Sep 28 05:11:09 dist-test-slave-6718 krb5kdc[4492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:11:09.754242 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:09.739117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.0.97.3:40153)
0928 05:11:09.739289 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:09.739296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:09.739400 (+ 104us) server_negotiation.cc:407] Connection header received
0928 05:11:09.739643 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:09.739648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:09.739708 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:09.739827 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:09.741048 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.741883 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.743205 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:09.743667 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:09.743802 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:09.748161 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:09.748171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:09.748186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:09.748217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:09.751418 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:09.752250 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:09.752257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:09.752259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:09.752398 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:09.752816 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:09.752819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:09.752821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:09.753156 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:09.753362 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:09.753765 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 05:11:09.753918 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:11:09.755019 4984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:09.871928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4862
I20260928 05:11:09.890722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 4505
I20260928 05:11:09.891726 4566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:09.996518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 4505
2026-09-28T05:11:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4330 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T05:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:10Z Disabled control of system clock
I20260928 05:11:10.060299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39641
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 05:11:10.372339 5041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:10.372617 5041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:10.372792 5041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:10.379701 5041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:10.379789 5041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:10.379884 5041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:10.379990 5041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:10.380074 5041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:10.388688 5041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:10.389907 5041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:10.391655 5041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:10.403244 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:10.403282 5047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:10.403245 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:10.404433 5041 server_base.cc:1034] running on GCE node
I20260928 05:11:10.405153 5041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:10.406476 5041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:10.407727 5041 hybrid_clock.cc:648] HybridClock initialized: now 1790572270407665 us; error 57 us; skew 500 ppm
I20260928 05:11:10.408059 5041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:10.410462 5041 webserver.cc:460] Webserver started at http://127.0.97.62:43565/ using document root <none> and password file <none>
I20260928 05:11:10.411162 5041 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:10.411274 5041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:10.411618 5041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:10.414350 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "411a31c12c15408cae26a543ac2bd01f"
format_stamp: "Formatted at 2026-09-28 05:11:10 on dist-test-slave-6718"
server_key: "3709e989520f369ff075f2ef13e69e04"
server_key_iv: "11b71d4d3a7541b533eea5c6cd28d3ea"
server_key_version: "encryptionkey@0"
I20260928 05:11:10.415186 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "411a31c12c15408cae26a543ac2bd01f"
format_stamp: "Formatted at 2026-09-28 05:11:10 on dist-test-slave-6718"
server_key: "3709e989520f369ff075f2ef13e69e04"
server_key_iv: "11b71d4d3a7541b533eea5c6cd28d3ea"
server_key_version: "encryptionkey@0"
I20260928 05:11:10.419971 5041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:11:10.423204 5055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:10.424436 5041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:10.424614 5041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "411a31c12c15408cae26a543ac2bd01f"
format_stamp: "Formatted at 2026-09-28 05:11:10 on dist-test-slave-6718"
server_key: "3709e989520f369ff075f2ef13e69e04"
server_key_iv: "11b71d4d3a7541b533eea5c6cd28d3ea"
server_key_version: "encryptionkey@0"
I20260928 05:11:10.424868 5041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:10.430388 5041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:10.431135 5041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:10.431421 5041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:10.449107 5041 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39641
I20260928 05:11:10.449138 5106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39641 every 8 connection(s)
I20260928 05:11:10.450388 5041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:10.453896 5107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:10.457207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5041
I20260928 05:11:10.457412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:10.458200 388 external_mini_cluster.cc:1428] Setting key 1d23c3a378251cb5da5fd8c539ccb42e
I20260928 05:11:10.466221 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:10.460238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.0.97.62:39641)
0928 05:11:10.461169 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:10.461179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:10.461218 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:11:10.462826 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:10.462847 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:10.463272 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:10.463668 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:10.464719 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:10.464725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:10.464737 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:10.464749 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:10.464801 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:10.464815 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:10.465038 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:10.465175 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 05:11:10.465403 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":331,"threads_started":1}
I20260928 05:11:10.466403 5107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f: Bootstrap starting.
I20260928 05:11:10.469530 5107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:10.470487 5107 log.cc:824] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:10.473003 5107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f: No bootstrap required, opened a new log
I20260928 05:11:10.480180 5107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411a31c12c15408cae26a543ac2bd01f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39641 } }
I20260928 05:11:10.480561 5107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:10.480659 5107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411a31c12c15408cae26a543ac2bd01f, State: Initialized, Role: FOLLOWER
I20260928 05:11:10.481197 5107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411a31c12c15408cae26a543ac2bd01f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39641 } }
I20260928 05:11:10.481427 5107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:10.481561 5107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:10.481734 5107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:10.483881 5107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411a31c12c15408cae26a543ac2bd01f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39641 } }
I20260928 05:11:10.484347 5107 leader_election.cc:302] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 411a31c12c15408cae26a543ac2bd01f; no voters:
I20260928 05:11:10.484874 5107 leader_election.cc:288] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:10.484978 5112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:10.485669 5112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [term 1 LEADER]: Becoming Leader. State: Replica: 411a31c12c15408cae26a543ac2bd01f, State: Running, Role: LEADER
I20260928 05:11:10.486148 5112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411a31c12c15408cae26a543ac2bd01f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39641 } }
I20260928 05:11:10.486861 5107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:10.488901 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "411a31c12c15408cae26a543ac2bd01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411a31c12c15408cae26a543ac2bd01f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39641 } } }
I20260928 05:11:10.489223 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:10.489599 5114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411a31c12c15408cae26a543ac2bd01f. Latest consensus state: current_term: 1 leader_uuid: "411a31c12c15408cae26a543ac2bd01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411a31c12c15408cae26a543ac2bd01f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39641 } } }
I20260928 05:11:10.489899 5114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:10.492194 5119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:10.496634 5119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:10.503917 5119 catalog_manager.cc:1347] Generated new cluster ID: 81d429e0220b4d6498cb5f578a6faac8
I20260928 05:11:10.504017 5119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:10.519778 5119 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:10.520190 5119 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:10.532234 5119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f: Generated new TSK 0
I20260928 05:11:10.533113 5119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:11:10.537640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39641
--builtin_ntp_servers=127.0.97.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 05:11:10.854475 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:10.854874 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:10.855051 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:10.862210 5131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:10.862361 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:10.862591 5131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:10.873494 5131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:10.875226 5131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:10.877313 5131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:10.890137 5136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:10.890242 5137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:10.890708 5139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:10.891031 5131 server_base.cc:1034] running on GCE node
I20260928 05:11:10.891616 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:10.892516 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:10.893754 5131 hybrid_clock.cc:648] HybridClock initialized: now 1790572270893713 us; error 54 us; skew 500 ppm
I20260928 05:11:10.894078 5131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:10.896245 5131 webserver.cc:460] Webserver started at http://127.0.97.1:46027/ using document root <none> and password file <none>
I20260928 05:11:10.896920 5131 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:10.897048 5131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:10.897361 5131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:10.900178 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "b01b8818b7594e39a66114655d6c86c1"
format_stamp: "Formatted at 2026-09-28 05:11:10 on dist-test-slave-6718"
server_key: "0d95a6e471306bfcf0c0509183769c75"
server_key_iv: "1a635ad83f2b21983fa49db5a171a8b8"
server_key_version: "encryptionkey@0"
I20260928 05:11:10.900946 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b01b8818b7594e39a66114655d6c86c1"
format_stamp: "Formatted at 2026-09-28 05:11:10 on dist-test-slave-6718"
server_key: "0d95a6e471306bfcf0c0509183769c75"
server_key_iv: "1a635ad83f2b21983fa49db5a171a8b8"
server_key_version: "encryptionkey@0"
I20260928 05:11:10.905855 5131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:11:10.909188 5145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:10.910496 5131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:11:10.910696 5131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "b01b8818b7594e39a66114655d6c86c1"
format_stamp: "Formatted at 2026-09-28 05:11:10 on dist-test-slave-6718"
server_key: "0d95a6e471306bfcf0c0509183769c75"
server_key_iv: "1a635ad83f2b21983fa49db5a171a8b8"
server_key_version: "encryptionkey@0"
I20260928 05:11:10.910878 5131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:10.916153 5131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:10.916837 5131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:10.917043 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:10.917829 5131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:10.919139 5131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:10.919222 5131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:10.919297 5131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:10.919350 5131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:10.951071 5131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34391
I20260928 05:11:10.951148 5257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34391 every 8 connection(s)
I20260928 05:11:10.952363 5131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:10.957001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5131
I20260928 05:11:10.957223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:10.957978 388 external_mini_cluster.cc:1428] Setting key 27bf8cce5b1a41d6daea7abba95cb65f
I20260928 05:11:10.960886 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:10.954825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60233 (local address 127.0.97.62:39641)
0928 05:11:10.954989 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:10.954993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:10.956330 (+ 1337us) server_negotiation.cc:407] Connection header received
0928 05:11:10.958731 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:10.958736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:10.958804 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:10.958901 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:10.960284 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:10.960288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:10.960292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:10.960306 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:10.960335 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:10.960345 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:10.960432 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:10.960663 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 05:11:10.960705 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:11:10.960960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39641
--builtin_ntp_servers=127.0.97.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 05:11:10.962206 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:10.955318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39641 (local address 127.0.97.1:60233)
0928 05:11:10.956185 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:10.956220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:10.958407 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:10.959150 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:10.959158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:10.959248 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:10.959952 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:10.959956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:10.960010 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:10.960115 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:10.960444 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:10.960454 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:10.960461 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:11:10.960743 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 05:11:10.960991 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":270,"threads_started":1}
I20260928 05:11:10.963608 5258 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39641
I20260928 05:11:10.963907 5258 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:10.964514 5258 heartbeater.cc:511] Master 127.0.97.62:39641 requested a full tablet report, sending...
I20260928 05:11:10.966827 5072 ts_manager.cc:194] Registered new tserver with Master: b01b8818b7594e39a66114655d6c86c1 (127.0.97.1:34391)
W20260928 05:11:11.273941 5261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:11.274228 5261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:11.274382 5261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:11.281554 5261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:11.281653 5261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:11.281836 5261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:11.292394 5261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:11.293723 5261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:11.295540 5261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:11.308676 5269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:11.308745 5266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:11.308725 5267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:11.314842 5261 server_base.cc:1034] running on GCE node
I20260928 05:11:11.315411 5261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:11.316265 5261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:11.317489 5261 hybrid_clock.cc:648] HybridClock initialized: now 1790572271317430 us; error 55 us; skew 500 ppm
I20260928 05:11:11.317808 5261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:11.320026 5261 webserver.cc:460] Webserver started at http://127.0.97.2:46193/ using document root <none> and password file <none>
I20260928 05:11:11.320698 5261 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:11.320824 5261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:11.321137 5261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:11.323956 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "e6b858dfff5b49fd889917dce935e750"
format_stamp: "Formatted at 2026-09-28 05:11:11 on dist-test-slave-6718"
server_key: "3df224e34e332a18fc06b6203d816dd7"
server_key_iv: "0baabe8267124cd50973648e1be7ee17"
server_key_version: "encryptionkey@0"
I20260928 05:11:11.324728 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e6b858dfff5b49fd889917dce935e750"
format_stamp: "Formatted at 2026-09-28 05:11:11 on dist-test-slave-6718"
server_key: "3df224e34e332a18fc06b6203d816dd7"
server_key_iv: "0baabe8267124cd50973648e1be7ee17"
server_key_version: "encryptionkey@0"
I20260928 05:11:11.329630 5261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:11:11.332818 5275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:11.333992 5261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:11.334172 5261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "e6b858dfff5b49fd889917dce935e750"
format_stamp: "Formatted at 2026-09-28 05:11:11 on dist-test-slave-6718"
server_key: "3df224e34e332a18fc06b6203d816dd7"
server_key_iv: "0baabe8267124cd50973648e1be7ee17"
server_key_version: "encryptionkey@0"
I20260928 05:11:11.334398 5261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:11.339733 5261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:11.340473 5261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:11.340739 5261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:11.341578 5261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:11.342979 5261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:11.343106 5261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:11.343247 5261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:11.343331 5261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:11.376029 5261 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40021
I20260928 05:11:11.376122 5387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40021 every 8 connection(s)
I20260928 05:11:11.377352 5261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:11.379973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5261
I20260928 05:11:11.380183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:11.380551 388 external_mini_cluster.cc:1428] Setting key 17d80ec964190032d62c9c0a17ab47fd
I20260928 05:11:11.383066 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39641
--builtin_ntp_servers=127.0.97.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 05:11:11.385744 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.379912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39695 (local address 127.0.97.62:39641)
0928 05:11:11.380131 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.380139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.381740 (+ 1601us) server_negotiation.cc:407] Connection header received
0928 05:11:11.383508 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.383513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.383575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.383680 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.385000 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.385005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.385010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.385028 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.385067 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.385078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.385195 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.385401 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.385464 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:11:11.386720 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.380497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39641 (local address 127.0.97.2:39695)
0928 05:11:11.381591 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.381634 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.383244 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.383918 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.383926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.384016 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.384720 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.384722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.384759 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.384839 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.385182 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.385193 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.385201 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.385470 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.385722 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":348,"threads_started":1}
I20260928 05:11:11.388038 5388 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39641
I20260928 05:11:11.388360 5388 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:11.388922 5388 heartbeater.cc:511] Master 127.0.97.62:39641 requested a full tablet report, sending...
I20260928 05:11:11.390308 5072 ts_manager.cc:194] Registered new tserver with Master: e6b858dfff5b49fd889917dce935e750 (127.0.97.2:40021)
W20260928 05:11:11.692898 5391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:11.693189 5391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:11.693279 5391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:11.700237 5391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:11.700328 5391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:11.700434 5391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:11.709313 5391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:11.710577 5391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:11.712412 5391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:11.725431 5396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:11.725409 5397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:11.725667 5399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:11.726150 5391 server_base.cc:1034] running on GCE node
I20260928 05:11:11.726843 5391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:11.727834 5391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:11.729077 5391 hybrid_clock.cc:648] HybridClock initialized: now 1790572271729028 us; error 36 us; skew 500 ppm
I20260928 05:11:11.729415 5391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:11.731645 5391 webserver.cc:460] Webserver started at http://127.0.97.3:45615/ using document root <none> and password file <none>
I20260928 05:11:11.732340 5391 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:11.732468 5391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:11.732787 5391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:11.735674 5391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "f8f4e195405847f78bdb74cbc635da35"
format_stamp: "Formatted at 2026-09-28 05:11:11 on dist-test-slave-6718"
server_key: "6b1eae5f7bb926f72df506fc5f44ba1f"
server_key_iv: "89a42d5014593f333e3cfa53c61f41fb"
server_key_version: "encryptionkey@0"
I20260928 05:11:11.736457 5391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f8f4e195405847f78bdb74cbc635da35"
format_stamp: "Formatted at 2026-09-28 05:11:11 on dist-test-slave-6718"
server_key: "6b1eae5f7bb926f72df506fc5f44ba1f"
server_key_iv: "89a42d5014593f333e3cfa53c61f41fb"
server_key_version: "encryptionkey@0"
I20260928 05:11:11.741415 5391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:11:11.744820 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:11.746114 5391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 05:11:11.746292 5391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "f8f4e195405847f78bdb74cbc635da35"
format_stamp: "Formatted at 2026-09-28 05:11:11 on dist-test-slave-6718"
server_key: "6b1eae5f7bb926f72df506fc5f44ba1f"
server_key_iv: "89a42d5014593f333e3cfa53c61f41fb"
server_key_version: "encryptionkey@0"
I20260928 05:11:11.746573 5391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:11.752146 5391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:11.752915 5391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:11.753197 5391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:11.754081 5391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:11.755496 5391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:11.755637 5391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:11.755764 5391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:11.755848 5391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:11.787760 5391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34319
I20260928 05:11:11.787843 5517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34319 every 8 connection(s)
I20260928 05:11:11.789037 5391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:11.791018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5391
I20260928 05:11:11.791221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:11.791725 388 external_mini_cluster.cc:1428] Setting key 4134847551930cdd07df2cd6756e9035
I20260928 05:11:11.797217 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.791736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59181 (local address 127.0.97.62:39641)
0928 05:11:11.792716 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.792724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.793497 (+ 773us) server_negotiation.cc:407] Connection header received
0928 05:11:11.795524 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.795528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.795575 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.795661 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.796721 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.796725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.796729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.796743 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.796772 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.796783 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.796873 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.796993 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.797035 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843}
I20260928 05:11:11.798152 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.792276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39641 (local address 127.0.97.3:59181)
0928 05:11:11.793354 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.793394 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.795306 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.795814 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.795821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.795905 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.796471 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.796473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.796508 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.796588 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.796846 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.796850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.796854 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.797038 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.797223 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":387,"threads_started":1}
I20260928 05:11:11.799423 5518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39641
I20260928 05:11:11.799706 5518 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:11.800215 5518 heartbeater.cc:511] Master 127.0.97.62:39641 requested a full tablet report, sending...
I20260928 05:11:11.801429 5072 ts_manager.cc:194] Registered new tserver with Master: f8f4e195405847f78bdb74cbc635da35 (127.0.97.3:34319)
I20260928 05:11:11.806706 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:11:11.813310 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.810757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.0.97.62:39641)
0928 05:11:11.810900 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.810904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.811512 (+ 608us) server_negotiation.cc:407] Connection header received
0928 05:11:11.811711 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.811715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.811759 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.811855 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.812723 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.812728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.812733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.812749 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.812785 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.812797 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.812912 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.812995 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.813049 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:11:11.816874 5072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:11.823429 5072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:11.847019 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.836561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.0.97.3:34319)
0928 05:11:11.843375 (+ 6814us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.843385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.843971 (+ 586us) server_negotiation.cc:407] Connection header received
0928 05:11:11.844053 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.844060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.844303 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.844461 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.846113 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.846120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.846132 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.846152 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.846198 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.846214 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.846363 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.846494 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.846611 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6656,"thread_start_us":6497,"threads_started":1}
I20260928 05:11:11.847038 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.834648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40021 (local address 127.0.0.1:48516)
0928 05:11:11.836531 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.836577 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.836726 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.844994 (+ 8268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.844998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.845022 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.845617 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.845620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.845632 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.845664 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.846216 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.846225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.846234 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.846419 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.846513 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"thread_start_us":2323,"threads_started":1}
I20260928 05:11:11.847097 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.836026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34319 (local address 127.0.0.1:33470)
0928 05:11:11.843760 (+ 7734us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.843776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.843862 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.844530 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.844534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.844552 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.845464 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.845467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.845498 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.845537 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.846346 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.846351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.846357 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.846510 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.846614 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7629,"mutex_wait_us":3,"thread_start_us":7492,"threads_started":1}
I20260928 05:11:11.847765 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.834804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47538)
0928 05:11:11.837669 (+ 2865us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.837692 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.837801 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.844282 (+ 6481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.844290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.844381 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.845254 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.845258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.845292 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.845379 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.846432 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.846436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.846440 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.846553 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.846597 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2417,"thread_start_us":1172,"threads_started":1}
I20260928 05:11:11.847915 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.835322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.97.1:34391)
0928 05:11:11.842967 (+ 7645us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.842978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.843012 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:11:11.843088 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.843097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.844001 (+ 904us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.844901 (+ 900us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.846203 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.846211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.846222 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.846238 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.846289 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.846316 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.846775 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.846839 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.846934 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7494,"thread_start_us":7382,"threads_started":1}
I20260928 05:11:11.848420 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.834834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.0.97.2:40021)
0928 05:11:11.844540 (+ 9706us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.844548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.844585 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:11:11.844645 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.844652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.844828 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.844977 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.845989 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.846000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.846013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.846030 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.846082 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.846106 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.846274 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.847745 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.848112 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9554,"thread_start_us":9442,"threads_started":1}
I20260928 05:11:11.851369 5323 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf1eb9cdc9c46538ac4324fac1e3489 (DEFAULT_TABLE table=test-table [id=72fb1001fca84f7296bcd6a9f0cb5f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:11.851205 5453 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf1eb9cdc9c46538ac4324fac1e3489 (DEFAULT_TABLE table=test-table [id=72fb1001fca84f7296bcd6a9f0cb5f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:11.851202 5193 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf1eb9cdc9c46538ac4324fac1e3489 (DEFAULT_TABLE table=test-table [id=72fb1001fca84f7296bcd6a9f0cb5f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:11.852574 5453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf1eb9cdc9c46538ac4324fac1e3489. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:11.852577 5323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf1eb9cdc9c46538ac4324fac1e3489. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:11.852715 5193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf1eb9cdc9c46538ac4324fac1e3489. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:11.862835 5536 tablet_bootstrap.cc:492] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Bootstrap starting.
I20260928 05:11:11.866169 5536 tablet_bootstrap.cc:654] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:11.866755 5537 tablet_bootstrap.cc:492] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Bootstrap starting.
I20260928 05:11:11.867446 5536 log.cc:824] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:11.868170 5538 tablet_bootstrap.cc:492] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Bootstrap starting.
I20260928 05:11:11.870225 5537 tablet_bootstrap.cc:654] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:11.871142 5538 tablet_bootstrap.cc:654] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:11.871388 5536 tablet_bootstrap.cc:492] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: No bootstrap required, opened a new log
I20260928 05:11:11.871464 5537 log.cc:824] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:11.871665 5536 ts_tablet_manager.cc:1397] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 05:11:11.872321 5538 log.cc:824] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:11.874480 5537 tablet_bootstrap.cc:492] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: No bootstrap required, opened a new log
I20260928 05:11:11.874761 5537 ts_tablet_manager.cc:1397] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260928 05:11:11.880934 5538 tablet_bootstrap.cc:492] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: No bootstrap required, opened a new log
I20260928 05:11:11.881233 5538 ts_tablet_manager.cc:1397] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260928 05:11:11.882462 5536 raft_consensus.cc:348] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.882962 5536 raft_consensus.cc:374] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:11.883095 5536 raft_consensus.cc:729] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b858dfff5b49fd889917dce935e750, State: Initialized, Role: FOLLOWER
I20260928 05:11:11.883750 5536 consensus_queue.cc:260] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.885340 5537 raft_consensus.cc:348] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.885908 5388 heartbeater.cc:503] Master 127.0.97.62:39641 was elected leader, sending a full tablet report...
I20260928 05:11:11.886709 5537 raft_consensus.cc:374] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:11.886886 5537 raft_consensus.cc:729] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4e195405847f78bdb74cbc635da35, State: Initialized, Role: FOLLOWER
I20260928 05:11:11.887252 5536 ts_tablet_manager.cc:1428] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260928 05:11:11.887758 5537 consensus_queue.cc:260] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.889493 5537 ts_tablet_manager.cc:1428] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20260928 05:11:11.889932 5518 heartbeater.cc:503] Master 127.0.97.62:39641 was elected leader, sending a full tablet report...
I20260928 05:11:11.890204 5538 raft_consensus.cc:348] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.890568 5538 raft_consensus.cc:374] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:11.890692 5538 raft_consensus.cc:729] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01b8818b7594e39a66114655d6c86c1, State: Initialized, Role: FOLLOWER
I20260928 05:11:11.891333 5538 consensus_queue.cc:260] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.892498 5258 heartbeater.cc:503] Master 127.0.97.62:39641 was elected leader, sending a full tablet report...
I20260928 05:11:11.893016 5538 ts_tablet_manager.cc:1428] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260928 05:11:11.925010 5542 raft_consensus.cc:482] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:11.925262 5542 raft_consensus.cc:504] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.926724 5542 leader_election.cc:288] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01b8818b7594e39a66114655d6c86c1 (127.0.97.1:34391), f8f4e195405847f78bdb74cbc635da35 (127.0.97.3:34319)
I20260928 05:11:11.930220 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.927133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45401 (local address 127.0.97.1:34391)
0928 05:11:11.927318 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.927324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.928045 (+ 721us) server_negotiation.cc:407] Connection header received
0928 05:11:11.928203 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.928208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.928269 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.928374 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.929579 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.929586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.929590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.929623 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.929661 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.929676 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.929802 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.929894 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.929952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:11:11.930220 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.927160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51633 (local address 127.0.97.3:34319)
0928 05:11:11.927322 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.927327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.928044 (+ 717us) server_negotiation.cc:407] Connection header received
0928 05:11:11.928216 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.928221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.928271 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.928374 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.929618 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.929623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.929627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.929646 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.929678 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.929692 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.929810 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.929894 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.929952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:11:11.930714 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.927022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34319 (local address 127.0.97.2:51633)
0928 05:11:11.927925 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.927946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.928102 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.928433 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.928437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.928451 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.929328 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.929333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.929357 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.929407 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.929780 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.929786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.929791 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.929921 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.929981 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":20,"spinlock_wait_cycles":23808,"thread_start_us":604,"threads_started":1}
I20260928 05:11:11.930735 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.926937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.97.2:45401)
0928 05:11:11.927889 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:11.927915 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:11.928049 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:11.928415 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:11.928420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:11.928445 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:11.929377 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:11:11.929380 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:11.929391 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:11:11.929416 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:11.929780 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:11.929786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:11.929791 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:11:11.929892 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:11:11.929968 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":655,"threads_started":1}
I20260928 05:11:11.931644 5213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf1eb9cdc9c46538ac4324fac1e3489" candidate_uuid: "e6b858dfff5b49fd889917dce935e750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01b8818b7594e39a66114655d6c86c1" is_pre_election: true
I20260928 05:11:11.931903 5473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf1eb9cdc9c46538ac4324fac1e3489" candidate_uuid: "e6b858dfff5b49fd889917dce935e750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4e195405847f78bdb74cbc635da35" is_pre_election: true
I20260928 05:11:11.932145 5213 raft_consensus.cc:2454] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b858dfff5b49fd889917dce935e750 in term 0.
I20260928 05:11:11.932432 5473 raft_consensus.cc:2454] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b858dfff5b49fd889917dce935e750 in term 0.
I20260928 05:11:11.932835 5277 leader_election.cc:302] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01b8818b7594e39a66114655d6c86c1, e6b858dfff5b49fd889917dce935e750; no voters:
I20260928 05:11:11.933301 5542 raft_consensus.cc:2781] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:11.933439 5542 raft_consensus.cc:482] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:11.933529 5542 raft_consensus.cc:3037] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:11.935774 5542 raft_consensus.cc:504] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.936492 5542 leader_election.cc:288] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [CANDIDATE]: Term 1 election: Requested vote from peers b01b8818b7594e39a66114655d6c86c1 (127.0.97.1:34391), f8f4e195405847f78bdb74cbc635da35 (127.0.97.3:34319)
I20260928 05:11:11.936830 5213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf1eb9cdc9c46538ac4324fac1e3489" candidate_uuid: "e6b858dfff5b49fd889917dce935e750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01b8818b7594e39a66114655d6c86c1"
I20260928 05:11:11.937075 5213 raft_consensus.cc:3037] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:11.936995 5473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf1eb9cdc9c46538ac4324fac1e3489" candidate_uuid: "e6b858dfff5b49fd889917dce935e750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4e195405847f78bdb74cbc635da35"
I20260928 05:11:11.937252 5473 raft_consensus.cc:3037] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:11.939565 5213 raft_consensus.cc:2454] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b858dfff5b49fd889917dce935e750 in term 1.
I20260928 05:11:11.939677 5473 raft_consensus.cc:2454] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b858dfff5b49fd889917dce935e750 in term 1.
I20260928 05:11:11.939934 5277 leader_election.cc:302] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b01b8818b7594e39a66114655d6c86c1, e6b858dfff5b49fd889917dce935e750; no voters:
I20260928 05:11:11.940222 5542 raft_consensus.cc:2781] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:11.940649 5542 raft_consensus.cc:686] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 1 LEADER]: Becoming Leader. State: Replica: e6b858dfff5b49fd889917dce935e750, State: Running, Role: LEADER
I20260928 05:11:11.941139 5542 consensus_queue.cc:237] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } }
I20260928 05:11:11.945562 5071 catalog_manager.cc:5495] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6b858dfff5b49fd889917dce935e750 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e6b858dfff5b49fd889917dce935e750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6b858dfff5b49fd889917dce935e750" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 } health_report { overall_health: UNKNOWN } } }
W20260928 05:11:11.954598 5259 tablet.cc:2369] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:11:11.965807 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:11.964064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.0.97.2:40021)
0928 05:11:11.964227 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:11.964231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:11.964250 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:11:11.964431 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:11.964435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:11.964506 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:11.964589 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:11.965368 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:11.965372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:11.965375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:11.965391 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:11.965420 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:11.965430 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:11.965527 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:11.965593 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 05:11:11.965638 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:11:11.973378 5473 raft_consensus.cc:1261] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 1 FOLLOWER]: Refusing update from remote peer e6b858dfff5b49fd889917dce935e750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:11.973440 5213 raft_consensus.cc:1261] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 1 FOLLOWER]: Refusing update from remote peer e6b858dfff5b49fd889917dce935e750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:11.974359 5547 consensus_queue.cc:1035] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01b8818b7594e39a66114655d6c86c1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:11.974716 5542 consensus_queue.cc:1035] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4e195405847f78bdb74cbc635da35" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:11.984335 5550 mvcc.cc:205] Tried to move back new op lower bound from 7334184025993945088 to 7334184025874841600. Current Snapshot: MvccSnapshot[applied={T|T < 7334184025993945088}]
I20260928 05:11:11.985556 5552 mvcc.cc:205] Tried to move back new op lower bound from 7334184025993945088 to 7334184025874841600. Current Snapshot: MvccSnapshot[applied={T|T < 7334184025993945088}]
I20260928 05:11:11.991074 5551 mvcc.cc:205] Tried to move back new op lower bound from 7334184025993945088 to 7334184025874841600. Current Snapshot: MvccSnapshot[applied={T|T < 7334184025993945088}]
I20260928 05:11:12.002769 5071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:12.003027 5071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:12.006163 5071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 411a31c12c15408cae26a543ac2bd01f: Sending DeleteTablet for 3 replicas of tablet 3bf1eb9cdc9c46538ac4324fac1e3489
I20260928 05:11:12.007205 5323 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf1eb9cdc9c46538ac4324fac1e3489 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:12 UTC) from {username='slave'} at 127.0.0.1:48516
I20260928 05:11:12.007313 5193 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf1eb9cdc9c46538ac4324fac1e3489 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:12 UTC) from {username='slave'} at 127.0.0.1:47538
I20260928 05:11:12.007973 5453 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf1eb9cdc9c46538ac4324fac1e3489 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:12 UTC) from {username='slave'} at 127.0.0.1:33470
I20260928 05:11:12.008883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5131
I20260928 05:11:12.014909 5561 tablet_replica.cc:330] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: stopping tablet replica
I20260928 05:11:12.015628 5561 raft_consensus.cc:2229] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:12.016804 5561 raft_consensus.cc:2258] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:12.018271 5560 tablet_replica.cc:330] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: stopping tablet replica
I20260928 05:11:12.018821 5560 raft_consensus.cc:2229] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:12.019599 5560 raft_consensus.cc:2258] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:12.019667 5561 ts_tablet_manager.cc:1905] T 3bf1eb9cdc9c46538ac4324fac1e3489 P b01b8818b7594e39a66114655d6c86c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:12.020182 5562 tablet_replica.cc:330] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: stopping tablet replica
I20260928 05:11:12.020555 5562 raft_consensus.cc:2229] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:12.020989 5562 raft_consensus.cc:2258] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:12.023241 5560 ts_tablet_manager.cc:1905] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:12.023902 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.010042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.97.1:34391)
0928 05:11:12.010241 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:12.010250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:12.014494 (+ 4244us) server_negotiation.cc:407] Connection header received
0928 05:11:12.014847 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:12.014857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:12.015035 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:12.015126 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:12.023044 (+ 7918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:12.023052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:12.023056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:12.023078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:12.023130 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:12.023146 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:12.023250 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:12.023506 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 05:11:12.023577 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:11:12.024917 5562 ts_tablet_manager.cc:1905] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:12.025028 5253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:12.034130 5560 ts_tablet_manager.cc:1918] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:12.034413 5560 log.cc:1196] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/3bf1eb9cdc9c46538ac4324fac1e3489
I20260928 05:11:12.035079 5560 ts_tablet_manager.cc:1939] T 3bf1eb9cdc9c46538ac4324fac1e3489 P e6b858dfff5b49fd889917dce935e750: Deleting consensus metadata
I20260928 05:11:12.035341 5562 ts_tablet_manager.cc:1918] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:12.035590 5562 log.cc:1196] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/3bf1eb9cdc9c46538ac4324fac1e3489
I20260928 05:11:12.036207 5562 ts_tablet_manager.cc:1939] T 3bf1eb9cdc9c46538ac4324fac1e3489 P f8f4e195405847f78bdb74cbc635da35: Deleting consensus metadata
I20260928 05:11:12.036970 5059 catalog_manager.cc:4841] TS e6b858dfff5b49fd889917dce935e750 (127.0.97.2:40021): tablet 3bf1eb9cdc9c46538ac4324fac1e3489 (table test-table [id=72fb1001fca84f7296bcd6a9f0cb5f17]) successfully deleted
I20260928 05:11:12.038210 5056 catalog_manager.cc:4841] TS f8f4e195405847f78bdb74cbc635da35 (127.0.97.3:34319): tablet 3bf1eb9cdc9c46538ac4324fac1e3489 (table test-table [id=72fb1001fca84f7296bcd6a9f0cb5f17]) successfully deleted
I20260928 05:11:12.120853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5131
I20260928 05:11:12.141024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5261
W20260928 05:11:12.141047 5057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34391 (error 108)
I20260928 05:11:12.143121 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.142585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47564)
0928 05:11:12.142815 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.142944 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260928 05:11:12.143361 5057 catalog_manager.cc:4584] TS b01b8818b7594e39a66114655d6c86c1 (127.0.97.1:34391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bf1eb9cdc9c46538ac4324fac1e3489: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
I20260928 05:11:12.144264 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.142145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.0.97.2:40021)
0928 05:11:12.142321 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:12.142327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:12.142353 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:11:12.142424 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:12.142428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:12.142475 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:12.142639 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:12.143663 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:12.143667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:12.143671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:12.143685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:12.143719 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:12.143731 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:12.143886 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:12.143944 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 05:11:12.144000 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:11:12.145077 5383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:12.196835 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.196305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47566)
0928 05:11:12.196530 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.196650 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 05:11:12.197829 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.197467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47578)
0928 05:11:12.197612 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.197691 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 05:11:12.252099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5261
I20260928 05:11:12.258241 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.257775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47592)
0928 05:11:12.257992 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.258101 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:11:12.259196 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.258875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47594)
0928 05:11:12.259025 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.259085 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 05:11:12.269688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5391
I20260928 05:11:12.272430 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.270386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.0.97.3:34319)
0928 05:11:12.270566 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:12.270572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:12.270592 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:11:12.270755 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:12.270759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:12.270829 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:12.270931 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:12.271812 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:12.271818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:12.271823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:12.271843 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:12.271877 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:12.271888 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:12.272015 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:12.272099 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 05:11:12.272162 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:11:12.273036 5513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:12.339723 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.339255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47608)
0928 05:11:12.339465 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.339556 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 05:11:12.340793 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.340427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34391 (local address 127.0.0.1:47620)
0928 05:11:12.340626 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:12.340688 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 05:11:12.361726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5391
I20260928 05:11:12.377854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5041
I20260928 05:11:12.379041 5102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:12.482332 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5041
2026-09-28T05:11:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2484 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 05:11:12.522433 388 mini_oidc.cc:152] Starting JWKS server
I20260928 05:11:12.524948 388 webserver.cc:460] Webserver started at https://127.0.0.1:37143/ using document root <none> and password file <none>
I20260928 05:11:12.525241 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:11:12.525835 388 webserver.cc:460] Webserver started at http://127.0.0.1:41861/ using document root <none> and password file <none>
2026-09-28T05:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:12Z Disabled control of system clock
I20260928 05:11:12.551126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38181
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38181
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37143/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/testchainca.cert with env {}
W20260928 05:11:12.859287 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:12.859544 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:12.859628 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:12.866477 5574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:12.866585 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:12.866621 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:12.866671 5574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:12.866716 5574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:11:12.866772 5574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:11:12.866802 5574 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37143/jwks.json
I20260928 05:11:12.876153 5574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37143/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:12.877293 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:12.878466 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:12.890101 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:12.890151 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:12.890197 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:12.896013 5574 server_base.cc:1034] running on GCE node
I20260928 05:11:12.896734 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:12.898027 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:12.899255 5574 hybrid_clock.cc:648] HybridClock initialized: now 1790572272899220 us; error 49 us; skew 500 ppm
I20260928 05:11:12.899588 5574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:12.901638 5574 webserver.cc:460] Webserver started at http://127.0.97.62:38611/ using document root <none> and password file <none>
I20260928 05:11:12.902293 5574 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:12.902419 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:12.902762 5574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:12.905463 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "c03b07a657b640eb97270b87e77f8168"
format_stamp: "Formatted at 2026-09-28 05:11:12 on dist-test-slave-6718"
server_key: "a22ecefa591c1e2a7317d4339f89f336"
server_key_iv: "6efe8e2c9d9b8e6c76a5e56965a9456b"
server_key_version: "encryptionkey@0"
I20260928 05:11:12.906240 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "c03b07a657b640eb97270b87e77f8168"
format_stamp: "Formatted at 2026-09-28 05:11:12 on dist-test-slave-6718"
server_key: "a22ecefa591c1e2a7317d4339f89f336"
server_key_iv: "6efe8e2c9d9b8e6c76a5e56965a9456b"
server_key_version: "encryptionkey@0"
I20260928 05:11:12.911105 5574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:11:12.914356 5588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:12.915643 5574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:12.915829 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "c03b07a657b640eb97270b87e77f8168"
format_stamp: "Formatted at 2026-09-28 05:11:12 on dist-test-slave-6718"
server_key: "a22ecefa591c1e2a7317d4339f89f336"
server_key_iv: "6efe8e2c9d9b8e6c76a5e56965a9456b"
server_key_version: "encryptionkey@0"
I20260928 05:11:12.916035 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:12.926100 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:12.926940 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:12.927220 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:12.944398 5574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38181
I20260928 05:11:12.944406 5639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38181 every 8 connection(s)
I20260928 05:11:12.945683 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:12.947088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5574
I20260928 05:11:12.947263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:12.948067 388 external_mini_cluster.cc:1428] Setting key 8804e4d073363400593dfe19b5a3d91c
I20260928 05:11:12.951243 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:12.960510 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:12.949704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.0.97.62:38181)
0928 05:11:12.950568 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:12.950597 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:12.950639 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:11:12.952227 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:12.952247 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:12.952659 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:12.953018 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:12.953589 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:12.954932 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:12.956217 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:12.956847 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:12.957037 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:12.958274 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:12.958282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:12.958294 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:12.958874 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:12.958947 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:12.958966 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:12.959116 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:12.959352 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 05:11:12.959638 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":271,"threads_started":1}
I20260928 05:11:12.961504 5640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168: Bootstrap starting.
I20260928 05:11:12.963871 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:11:12.964687 5640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:12.965603 5640 log.cc:824] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:12.968333 5640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168: No bootstrap required, opened a new log
I20260928 05:11:12.975253 5640 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b07a657b640eb97270b87e77f8168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38181 } }
I20260928 05:11:12.975622 5640 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:12.975728 5640 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03b07a657b640eb97270b87e77f8168, State: Initialized, Role: FOLLOWER
I20260928 05:11:12.976262 5640 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b07a657b640eb97270b87e77f8168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38181 } }
I20260928 05:11:12.976492 5640 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:12.976634 5640 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:12.976804 5640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:12.978955 5640 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b07a657b640eb97270b87e77f8168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38181 } }
I20260928 05:11:12.979465 5640 leader_election.cc:302] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03b07a657b640eb97270b87e77f8168; no voters:
I20260928 05:11:12.980007 5640 leader_election.cc:288] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:12.980130 5645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:12.980839 5645 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [term 1 LEADER]: Becoming Leader. State: Replica: c03b07a657b640eb97270b87e77f8168, State: Running, Role: LEADER
I20260928 05:11:12.981379 5645 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b07a657b640eb97270b87e77f8168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38181 } }
I20260928 05:11:12.982023 5640 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:12.984120 5647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03b07a657b640eb97270b87e77f8168. Latest consensus state: current_term: 1 leader_uuid: "c03b07a657b640eb97270b87e77f8168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b07a657b640eb97270b87e77f8168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38181 } } }
I20260928 05:11:12.984565 5647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:12.985098 5646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03b07a657b640eb97270b87e77f8168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03b07a657b640eb97270b87e77f8168" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38181 } } }
I20260928 05:11:12.985446 5646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:12.986512 5653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:12.991443 5653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:13.000047 5653 catalog_manager.cc:1347] Generated new cluster ID: 6267256b37944a00a088417879180715
I20260928 05:11:13.000152 5653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:13.038856 5653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:13.040211 5653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:13.051677 5653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03b07a657b640eb97270b87e77f8168: Generated new TSK 0
I20260928 05:11:13.052528 5653 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 05:11:13.093144 5670 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 05:11:13.093314 5670 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:11:13.094452 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:13.071757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.97.62:38181)
0928 05:11:13.071944 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:13.071951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:13.072531 (+ 580us) server_negotiation.cc:407] Connection header received
0928 05:11:13.072841 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:13.072853 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:13.072947 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:13.073113 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:11:13.073596 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.074734 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.076153 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.076862 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.076994 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:13.077556 (+ 562us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:11:13.093782 (+ 16226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 05:11:13.094136 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 05:11:13.094609 5643 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 05:11:13.095764 5669 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:13.071742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38181 (local address 127.0.0.1:44018)
0928 05:11:13.072426 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:13.072443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:13.072620 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:13.073133 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:13.073136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:13.073146 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:11:13.073453 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:13.073461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.074868 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.074872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:13.076015 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:13.076027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.077009 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.077069 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:13.077219 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:13.077275 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:11:13.094041 (+ 16766us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 05:11:13.094315 (+ 274us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":356,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:38181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37143/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 05:11:13.097393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5574
I20260928 05:11:13.098693 5635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:13.192047 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5574
2026-09-28T05:11:13Z chronyd exiting
I20260928 05:11:13.242374 388 test_util.cc:175] -----------------------------------------------
I20260928 05:11:13.242496 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790572228671169-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 05:11:13.246042 388 mini_oidc.cc:152] Starting JWKS server
I20260928 05:11:13.247834 388 webserver.cc:460] Webserver started at https://127.0.0.1:44647/ using document root <none> and password file <none>
I20260928 05:11:13.248085 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:11:13.248697 388 webserver.cc:460] Webserver started at http://127.0.0.1:45677/ using document root <none> and password file <none>
2026-09-28T05:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:13Z Disabled control of system clock
I20260928 05:11:13.273744 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39539
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44647/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/testchainca.cert with env {}
W20260928 05:11:13.582883 5677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:13.583204 5677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:13.583298 5677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:13.590351 5677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:13.590446 5677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:13.590484 5677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:13.590556 5677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:13.590601 5677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:11:13.590663 5677 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:11:13.590703 5677 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44647/jwks.json
I20260928 05:11:13.600711 5677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44647/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:13.601912 5677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:13.603137 5677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:13.614630 5682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:13.614619 5685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:13.614645 5683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:13.615600 5677 server_base.cc:1034] running on GCE node
I20260928 05:11:13.616379 5677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:13.617727 5677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:13.618948 5677 hybrid_clock.cc:648] HybridClock initialized: now 1790572273618898 us; error 63 us; skew 500 ppm
I20260928 05:11:13.619302 5677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:13.621487 5677 webserver.cc:460] Webserver started at http://127.0.97.62:41257/ using document root <none> and password file <none>
I20260928 05:11:13.622260 5677 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:13.622392 5677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:13.622780 5677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:13.625676 5677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "846203bb012d4f2aad6c9d32677e7d74"
format_stamp: "Formatted at 2026-09-28 05:11:13 on dist-test-slave-6718"
server_key: "5c36f63aaa8a1c044e28e842cae3bdad"
server_key_iv: "531a79f357ea9003c536e9acff5a739d"
server_key_version: "encryptionkey@0"
I20260928 05:11:13.626487 5677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "846203bb012d4f2aad6c9d32677e7d74"
format_stamp: "Formatted at 2026-09-28 05:11:13 on dist-test-slave-6718"
server_key: "5c36f63aaa8a1c044e28e842cae3bdad"
server_key_iv: "531a79f357ea9003c536e9acff5a739d"
server_key_version: "encryptionkey@0"
I20260928 05:11:13.631518 5677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:11:13.634896 5691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:13.636228 5677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:11:13.636417 5677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "846203bb012d4f2aad6c9d32677e7d74"
format_stamp: "Formatted at 2026-09-28 05:11:13 on dist-test-slave-6718"
server_key: "5c36f63aaa8a1c044e28e842cae3bdad"
server_key_iv: "531a79f357ea9003c536e9acff5a739d"
server_key_version: "encryptionkey@0"
I20260928 05:11:13.636627 5677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:13.667579 5677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:13.668387 5677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:13.668665 5677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:13.686432 5677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39539
I20260928 05:11:13.686452 5742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39539 every 8 connection(s)
I20260928 05:11:13.687777 5677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:13.689894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5677
I20260928 05:11:13.690120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:13.690924 388 external_mini_cluster.cc:1428] Setting key 761cdc1080a0362e6402c268e0c99787
I20260928 05:11:13.692306 5743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:13.703866 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:13.692914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.0.97.62:39539)
0928 05:11:13.693895 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:13.693905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:13.693945 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:11:13.695432 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:13.695452 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:13.695906 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:13.696309 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:13.696856 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.698512 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.699774 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.700414 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.700603 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:13.701731 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:13.701739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:13.701755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:13.702295 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:13.702366 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:13.702388 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:13.702523 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:13.702746 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 05:11:13.703015 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":405,"threads_started":1}
I20260928 05:11:13.704775 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74: Bootstrap starting.
I20260928 05:11:13.707453 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:11:13.708045 5743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:13.708961 5743 log.cc:824] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:13.711561 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74: No bootstrap required, opened a new log
I20260928 05:11:13.718219 5743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846203bb012d4f2aad6c9d32677e7d74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39539 } }
I20260928 05:11:13.718542 5743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:13.718662 5743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846203bb012d4f2aad6c9d32677e7d74, State: Initialized, Role: FOLLOWER
I20260928 05:11:13.719146 5743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846203bb012d4f2aad6c9d32677e7d74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39539 } }
I20260928 05:11:13.719354 5743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:13.719506 5743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:13.719673 5743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:13.721655 5743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846203bb012d4f2aad6c9d32677e7d74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39539 } }
I20260928 05:11:13.722169 5743 leader_election.cc:302] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846203bb012d4f2aad6c9d32677e7d74; no voters:
I20260928 05:11:13.722751 5743 leader_election.cc:288] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:13.722888 5748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:13.723968 5748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [term 1 LEADER]: Becoming Leader. State: Replica: 846203bb012d4f2aad6c9d32677e7d74, State: Running, Role: LEADER
I20260928 05:11:13.724737 5743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:13.724682 5748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846203bb012d4f2aad6c9d32677e7d74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39539 } }
I20260928 05:11:13.731060 5749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846203bb012d4f2aad6c9d32677e7d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846203bb012d4f2aad6c9d32677e7d74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39539 } } }
I20260928 05:11:13.730904 5750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846203bb012d4f2aad6c9d32677e7d74. Latest consensus state: current_term: 1 leader_uuid: "846203bb012d4f2aad6c9d32677e7d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846203bb012d4f2aad6c9d32677e7d74" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39539 } } }
I20260928 05:11:13.731384 5749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:13.731433 5750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74 [sys.catalog]: This master's current role is: LEADER
W20260928 05:11:13.740562 5762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:11:13.740748 5762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:11:13.744232 5763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:13.745788 5763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:13.753274 5763 catalog_manager.cc:1347] Generated new cluster ID: 96aeb6de5587485c9200e35f1d4ba957
I20260928 05:11:13.753387 5763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:13.767812 5763 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:13.769157 5763 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:13.785616 5763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846203bb012d4f2aad6c9d32677e7d74: Generated new TSK 0
I20260928 05:11:13.786419 5763 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 05:11:13.827363 5773 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 05:11:13.827486 5773 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:11:13.828917 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:13.808957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.0.97.62:39539)
0928 05:11:13.809181 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:13.809191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:13.809595 (+ 404us) server_negotiation.cc:407] Connection header received
0928 05:11:13.809830 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:13.809833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:13.809921 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:13.810034 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:11:13.810577 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.811317 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.813317 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.813738 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.813826 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:13.814373 (+ 547us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:11:13.828118 (+ 13745us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 05:11:13.828524 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 05:11:13.829027 5746 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 05:11:13.829778 5772 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:13.808863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39539 (local address 127.0.0.1:38634)
0928 05:11:13.809483 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:13.809507 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:13.809652 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:13.810081 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:13.810086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:13.810099 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:11:13.810380 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:13.810391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.811458 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.811464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:13.813174 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:13.813190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:13.813853 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:13.813897 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:13.814103 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:13.814147 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:11:13.829303 (+ 15156us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 05:11:13.829501 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":327,"threads_started":1}
I20260928 05:11:13.830936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5677
I20260928 05:11:13.832103 5738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:13.925881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5677
2026-09-28T05:11:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (734 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 05:11:13.980772 388 mini_oidc.cc:152] Starting JWKS server
I20260928 05:11:13.982600 388 webserver.cc:460] Webserver started at https://127.0.0.1:40899/ using document root <none> and password file <none>
I20260928 05:11:13.982824 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 05:11:13.983454 388 webserver.cc:460] Webserver started at http://127.0.0.1:36151/ using document root <none> and password file <none>
2026-09-28T05:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:13Z Disabled control of system clock
I20260928 05:11:14.009001 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39267
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39267
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40899/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 05:11:14.325470 5780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:14.325779 5780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:14.325937 5780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:14.333041 5780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:14.333204 5780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:14.333293 5780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:14.333396 5780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:14.333478 5780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:11:14.333562 5780 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:11:14.333640 5780 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40899/jwks.json
I20260928 05:11:14.342424 5780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40899/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:14.343658 5780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:14.345463 5780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:14.356668 5785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:14.356703 5786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:14.356760 5788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:14.357573 5780 server_base.cc:1034] running on GCE node
I20260928 05:11:14.358367 5780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:14.359723 5780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:14.361006 5780 hybrid_clock.cc:648] HybridClock initialized: now 1790572274360920 us; error 52 us; skew 500 ppm
I20260928 05:11:14.361356 5780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:14.368006 5780 webserver.cc:460] Webserver started at http://127.0.97.62:46469/ using document root <none> and password file <none>
I20260928 05:11:14.368670 5780 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:14.368799 5780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:14.369153 5780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:14.371997 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "79861beaa59e4d30a81e53054fe78dc1"
format_stamp: "Formatted at 2026-09-28 05:11:14 on dist-test-slave-6718"
server_key: "9228e2cbf2a4752dc4d448e36c4c72d2"
server_key_iv: "8b146d02a18e3d0c7d38ead7b91412ac"
server_key_version: "encryptionkey@0"
I20260928 05:11:14.372764 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "79861beaa59e4d30a81e53054fe78dc1"
format_stamp: "Formatted at 2026-09-28 05:11:14 on dist-test-slave-6718"
server_key: "9228e2cbf2a4752dc4d448e36c4c72d2"
server_key_iv: "8b146d02a18e3d0c7d38ead7b91412ac"
server_key_version: "encryptionkey@0"
I20260928 05:11:14.377645 5780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:11:14.380918 5794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:14.382115 5780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:14.382304 5780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "79861beaa59e4d30a81e53054fe78dc1"
format_stamp: "Formatted at 2026-09-28 05:11:14 on dist-test-slave-6718"
server_key: "9228e2cbf2a4752dc4d448e36c4c72d2"
server_key_iv: "8b146d02a18e3d0c7d38ead7b91412ac"
server_key_version: "encryptionkey@0"
I20260928 05:11:14.382510 5780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:14.400777 5780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:14.401593 5780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:14.401877 5780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:14.419885 5780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39267
I20260928 05:11:14.419893 5845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39267 every 8 connection(s)
I20260928 05:11:14.421265 5780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:14.425020 5846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:14.426391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5780
I20260928 05:11:14.426625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:14.427250 388 external_mini_cluster.cc:1428] Setting key b802c8e1d88e5f07eefe62c9466658f8
I20260928 05:11:14.436702 5846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1: Bootstrap starting.
I20260928 05:11:14.440294 5846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:14.440649 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:14.429045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.0.97.62:39267)
0928 05:11:14.430066 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:14.430076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:14.430116 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:11:14.431835 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:14.431857 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:14.432305 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:14.432621 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:14.433244 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:14.434578 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:14.436313 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:14.436989 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:14.437109 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:14.438262 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:14.438269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:14.438284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:14.438869 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:14.438962 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:14.438982 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:14.439071 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:14.439311 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 05:11:14.439644 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":414,"threads_started":1}
I20260928 05:11:14.441596 5846 log.cc:824] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:14.444324 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:11:14.444620 5846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1: No bootstrap required, opened a new log
I20260928 05:11:14.451706 5846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79861beaa59e4d30a81e53054fe78dc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39267 } }
I20260928 05:11:14.452122 5846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:14.452263 5846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79861beaa59e4d30a81e53054fe78dc1, State: Initialized, Role: FOLLOWER
I20260928 05:11:14.452790 5846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79861beaa59e4d30a81e53054fe78dc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39267 } }
I20260928 05:11:14.453020 5846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:14.453155 5846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:14.453320 5846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:14.455411 5846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79861beaa59e4d30a81e53054fe78dc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39267 } }
I20260928 05:11:14.455886 5846 leader_election.cc:302] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79861beaa59e4d30a81e53054fe78dc1; no voters:
I20260928 05:11:14.456408 5846 leader_election.cc:288] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:14.456588 5851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:14.457847 5851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [term 1 LEADER]: Becoming Leader. State: Replica: 79861beaa59e4d30a81e53054fe78dc1, State: Running, Role: LEADER
I20260928 05:11:14.458338 5851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79861beaa59e4d30a81e53054fe78dc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39267 } }
I20260928 05:11:14.458602 5846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:14.461325 5852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79861beaa59e4d30a81e53054fe78dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79861beaa59e4d30a81e53054fe78dc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39267 } } }
I20260928 05:11:14.461207 5853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79861beaa59e4d30a81e53054fe78dc1. Latest consensus state: current_term: 1 leader_uuid: "79861beaa59e4d30a81e53054fe78dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79861beaa59e4d30a81e53054fe78dc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39267 } } }
I20260928 05:11:14.461802 5852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:14.461855 5853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:14.469128 5860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:14.474159 5860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:14.481915 5860 catalog_manager.cc:1347] Generated new cluster ID: 7c55ce64dbf24a98aa2db683ab12398a
I20260928 05:11:14.482016 5860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:14.494096 5860 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:14.495396 5860 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:14.511186 5860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79861beaa59e4d30a81e53054fe78dc1: Generated new TSK 0
I20260928 05:11:14.512084 5860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:11:14.539167 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:14.530833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.0.97.62:39267)
0928 05:11:14.531033 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:14.531045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:14.531589 (+ 544us) server_negotiation.cc:407] Connection header received
0928 05:11:14.531757 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:14.531763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:14.531855 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:14.532028 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:11:14.532468 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:14.533631 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:14.534970 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:14.535680 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:14.535809 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:14.536228 (+ 419us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:11:14.538505 (+ 2277us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40899/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 05:11:14.538860 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40899/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 05:11:14.539292 5849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40899/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 05:11:14.540771 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:14.530834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39267 (local address 127.0.0.1:35966)
0928 05:11:14.531471 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:14.531486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:14.531599 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:14.532030 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:14.532033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:14.532042 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:11:14.532313 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:14.532321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:14.533752 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:14.533756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:14.534845 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:14.534856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:14.535787 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:14.535844 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:14.535958 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:14.535988 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:11:14.538729 (+ 2741us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40899/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 05:11:14.540387 (+ 1658us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40899/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":1390,"thread_start_us":366,"threads_started":1}
I20260928 05:11:14.542178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5780
I20260928 05:11:14.543650 5841 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 05:11:14.549611 5876 webserver.cc:533] Webserver: sslize failed:
E20260928 05:11:14.549721 5876 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 05:11:14.639678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5780
2026-09-28T05:11:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T05:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:14Z Disabled control of system clock
I20260928 05:11:14.719239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42523
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42523
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/testchainca.cert with env {}
W20260928 05:11:15.030050 5881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:15.030354 5881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:15.030443 5881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:15.037591 5881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:15.037698 5881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:15.037736 5881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:15.037784 5881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:15.037820 5881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 05:11:15.037879 5881 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 05:11:15.037918 5881 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 05:11:15.047559 5881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:15.048717 5881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:15.049881 5881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:15.060926 5886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:15.060889 5889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:15.060957 5887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:15.061782 5881 server_base.cc:1034] running on GCE node
I20260928 05:11:15.062479 5881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:15.063732 5881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:15.064950 5881 hybrid_clock.cc:648] HybridClock initialized: now 1790572275064910 us; error 53 us; skew 500 ppm
I20260928 05:11:15.065279 5881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:15.067342 5881 webserver.cc:460] Webserver started at http://127.0.97.62:37053/ using document root <none> and password file <none>
I20260928 05:11:15.067968 5881 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:15.068104 5881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:15.068393 5881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:15.071084 5881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "4ba1155088124df3a61d3b42fd4fa8d8"
format_stamp: "Formatted at 2026-09-28 05:11:15 on dist-test-slave-6718"
server_key: "ecf67771bbb33e21963e8c56b00a4058"
server_key_iv: "f110c4d6157e4ef021097c0c161abbaf"
server_key_version: "encryptionkey@0"
I20260928 05:11:15.071842 5881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "4ba1155088124df3a61d3b42fd4fa8d8"
format_stamp: "Formatted at 2026-09-28 05:11:15 on dist-test-slave-6718"
server_key: "ecf67771bbb33e21963e8c56b00a4058"
server_key_iv: "f110c4d6157e4ef021097c0c161abbaf"
server_key_version: "encryptionkey@0"
I20260928 05:11:15.076534 5881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:11:15.079632 5895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:15.080899 5881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:15.081066 5881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "4ba1155088124df3a61d3b42fd4fa8d8"
format_stamp: "Formatted at 2026-09-28 05:11:15 on dist-test-slave-6718"
server_key: "ecf67771bbb33e21963e8c56b00a4058"
server_key_iv: "f110c4d6157e4ef021097c0c161abbaf"
server_key_version: "encryptionkey@0"
I20260928 05:11:15.081279 5881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:15.108224 5881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:15.109014 5881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:15.109287 5881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:15.126876 5881 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42523
I20260928 05:11:15.126895 5946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42523 every 8 connection(s)
I20260928 05:11:15.128229 5881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:15.131942 5947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:15.136853 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 5881
I20260928 05:11:15.137074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:15.137830 388 external_mini_cluster.cc:1428] Setting key c6dc5d5b9199140bbc14a67c9a206a72
I20260928 05:11:15.143240 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8: Bootstrap starting.
I20260928 05:11:15.146736 5947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:15.147812 5947 log.cc:824] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:15.150895 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8: No bootstrap required, opened a new log
I20260928 05:11:15.151374 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:15.139684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.0.97.62:42523)
0928 05:11:15.140688 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:15.140701 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:15.140741 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:11:15.142338 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:15.142359 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:15.142920 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:15.143419 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:15.144048 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:15.145325 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:15.147152 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:15.147778 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:15.147899 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:15.149149 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:15.149160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:15.149183 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:15.149691 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:15.149743 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:15.149762 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:15.149850 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:15.150063 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 05:11:15.150381 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":398,"threads_started":1}
I20260928 05:11:15.155089 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:11:15.158869 5947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42523 } }
I20260928 05:11:15.159178 5947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:15.159258 5947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba1155088124df3a61d3b42fd4fa8d8, State: Initialized, Role: FOLLOWER
I20260928 05:11:15.159782 5947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42523 } }
I20260928 05:11:15.159974 5947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:15.160065 5947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:15.160187 5947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:15.162438 5947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42523 } }
I20260928 05:11:15.162956 5947 leader_election.cc:302] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ba1155088124df3a61d3b42fd4fa8d8; no voters:
I20260928 05:11:15.163472 5947 leader_election.cc:288] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:15.163679 5952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:15.165387 5947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:15.165766 5952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [term 1 LEADER]: Becoming Leader. State: Replica: 4ba1155088124df3a61d3b42fd4fa8d8, State: Running, Role: LEADER
I20260928 05:11:15.166296 5952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42523 } }
I20260928 05:11:15.168949 5953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42523 } } }
I20260928 05:11:15.169064 5954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba1155088124df3a61d3b42fd4fa8d8. Latest consensus state: current_term: 1 leader_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba1155088124df3a61d3b42fd4fa8d8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42523 } } }
I20260928 05:11:15.169327 5953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:15.169409 5954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:15.175705 5961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:15.179565 5961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:15.187815 5961 catalog_manager.cc:1347] Generated new cluster ID: 2f8a2a9623274367b29d81834491c67d
I20260928 05:11:15.187912 5961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:15.201463 5961 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:15.203291 5961 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:15.209988 5961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ba1155088124df3a61d3b42fd4fa8d8: Generated new TSK 0
I20260928 05:11:15.210821 5961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:11:15.283443 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:15.229779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.97.62:42523)
0928 05:11:15.230011 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:15.230019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:15.230368 (+ 349us) server_negotiation.cc:407] Connection header received
0928 05:11:15.230649 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:15.230653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:15.230723 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:15.230851 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0928 05:11:15.231398 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:15.232108 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:15.234103 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:15.234587 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:15.234699 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:15.235265 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 05:11:15.282884 (+ 47619us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 05:11:15.283212 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 05:11:15.283612 5950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 05:11:15.283918 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:15.229598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42523 (local address 127.0.0.1:55710)
0928 05:11:15.230240 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:15.230260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:15.230416 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:15.230902 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:15.230908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:15.230920 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 05:11:15.231243 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:15.231257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:15.232258 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:15.232266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:15.233944 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:15.233959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:15.234716 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:15.234771 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:15.234965 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:15.235010 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 05:11:15.283195 (+ 48185us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 05:11:15.283532 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":340,"threads_started":1}
I20260928 05:11:15.285136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 5881
I20260928 05:11:15.286428 5942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:15.382298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 5881
2026-09-28T05:11:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (722 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1504 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (400 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:17 dist-test-slave-6718 krb5kdc[5992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:17 dist-test-slave-6718 krb5kdc[5992](info): set up 2 sockets
Sep 28 05:11:17 dist-test-slave-6718 krb5kdc[5992](info): commencing operation
krb5kdc: starting...
W20260928 05:11:18.379233 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:18 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:11:18.577394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37195
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:18.889972 6005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:18.890326 6005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:18.890492 6005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:18.897682 6005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:18.897830 6005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:18.897943 6005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:18.898058 6005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:18.898144 6005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:18.907889 6005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:18.909178 6005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:18.910477 6005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:18.921936 6013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:18.922128 6010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:18.922097 6011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:18.923008 6005 server_base.cc:1034] running on GCE node
I20260928 05:11:18.923712 6005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:18.924999 6005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:18.926254 6005 hybrid_clock.cc:648] HybridClock initialized: now 1790572278926167 us; error 75 us; skew 500 ppm
I20260928 05:11:18.926671 6005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:18 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:18.929747 6005 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:11:18.931254 6005 webserver.cc:460] Webserver started at http://127.0.97.62:38625/ using document root <none> and password file <none>
I20260928 05:11:18.931903 6005 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:18.932041 6005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:18.932341 6005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:18.935235 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "6d1b03f889054c469419131ca42600be"
format_stamp: "Formatted at 2026-09-28 05:11:18 on dist-test-slave-6718"
server_key: "3971936ea6a4980739c07def043d4bd3"
server_key_iv: "16ed12c5dc520d9f1c14aba1fbb0702b"
server_key_version: "encryptionkey@0"
I20260928 05:11:18.935984 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d1b03f889054c469419131ca42600be"
format_stamp: "Formatted at 2026-09-28 05:11:18 on dist-test-slave-6718"
server_key: "3971936ea6a4980739c07def043d4bd3"
server_key_iv: "16ed12c5dc520d9f1c14aba1fbb0702b"
server_key_version: "encryptionkey@0"
I20260928 05:11:18.940997 6005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:11:18.944408 6020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:18.945787 6005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:11:18.945963 6005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "6d1b03f889054c469419131ca42600be"
format_stamp: "Formatted at 2026-09-28 05:11:18 on dist-test-slave-6718"
server_key: "3971936ea6a4980739c07def043d4bd3"
server_key_iv: "16ed12c5dc520d9f1c14aba1fbb0702b"
server_key_version: "encryptionkey@0"
I20260928 05:11:18.946174 6005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:18.959201 6005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:18.963425 6005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:18.963706 6005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:18.981406 6005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37195
I20260928 05:11:18.981436 6071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37195 every 8 connection(s)
I20260928 05:11:18.982756 6005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:18.985216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6005
I20260928 05:11:18.985433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:18.986028 388 external_mini_cluster.cc:1428] Setting key 135bb9448c8eb22d13ea57c52e1761f9
I20260928 05:11:18.986474 6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:19.000619 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be: Bootstrap starting.
Sep 28 05:11:18 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:19.003756 6072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:19.004781 6072 log.cc:824] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:19.007431 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be: No bootstrap required, opened a new log
I20260928 05:11:19.008718 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:18.988050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.0.97.62:37195)
0928 05:11:18.988795 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:18.988807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:18.988848 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:11:18.990871 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:18.990893 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:18.991288 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:18.991614 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:18.993293 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:18.994658 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:18.996452 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:18.997160 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:18.997298 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:19.001812 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:19.001820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:19.001834 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:19.001872 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:19.004974 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:19.005623 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:19.005631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:19.005646 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:19.005809 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:19.006177 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:19.006182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:19.006185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:19.006715 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:19.007008 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:19.007416 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 05:11:19.007821 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":350,"threads_started":1}
I20260928 05:11:19.015296 6072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b03f889054c469419131ca42600be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260928 05:11:19.015676 6072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:19.015781 6072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1b03f889054c469419131ca42600be, State: Initialized, Role: FOLLOWER
I20260928 05:11:19.016328 6072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b03f889054c469419131ca42600be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260928 05:11:19.016559 6072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:19.016697 6072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:19.016863 6072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:19.019135 6072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b03f889054c469419131ca42600be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260928 05:11:19.019627 6072 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1b03f889054c469419131ca42600be; no voters:
I20260928 05:11:19.020314 6072 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:19.020430 6077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:19.021293 6077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [term 1 LEADER]: Becoming Leader. State: Replica: 6d1b03f889054c469419131ca42600be, State: Running, Role: LEADER
I20260928 05:11:19.021795 6077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b03f889054c469419131ca42600be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } }
I20260928 05:11:19.022589 6072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:19.024763 6078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1b03f889054c469419131ca42600be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b03f889054c469419131ca42600be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } } }
I20260928 05:11:19.025347 6078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:19.025234 6079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1b03f889054c469419131ca42600be. Latest consensus state: current_term: 1 leader_uuid: "6d1b03f889054c469419131ca42600be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1b03f889054c469419131ca42600be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37195 } } }
I20260928 05:11:19.025493 6079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:19.031980 6086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:19.038143 6086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:19.047979 6086 catalog_manager.cc:1347] Generated new cluster ID: 48cdba517f3a4c6ab3d29865d598b83a
I20260928 05:11:19.048170 6086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:19.066864 6086 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:19.068104 6086 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:19.075757 6086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1b03f889054c469419131ca42600be: Generated new TSK 0
I20260928 05:11:19.076527 6086 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:11:19.133612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:19.485976 6098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:19.486248 6098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:19.486339 6098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:19.493394 6098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:19.493503 6098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:19.493611 6098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:19.503315 6098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:19.504595 6098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:19.505908 6098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:19.518940 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:19.518904 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:19.518975 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:19.519549 6098 server_base.cc:1034] running on GCE node
I20260928 05:11:19.520220 6098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:19.521215 6098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:19.522416 6098 hybrid_clock.cc:648] HybridClock initialized: now 1790572279522363 us; error 51 us; skew 500 ppm
I20260928 05:11:19.522768 6098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:19 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:19.529064 6098 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:11:19.530424 6098 webserver.cc:460] Webserver started at http://127.0.97.1:39519/ using document root <none> and password file <none>
I20260928 05:11:19.531195 6098 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:19.531349 6098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:19.531656 6098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:19.534668 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "511e83c9a31f4e64a55976254163f5c7"
format_stamp: "Formatted at 2026-09-28 05:11:19 on dist-test-slave-6718"
server_key: "e21a605ec2dd9bdd43b39a7712de8396"
server_key_iv: "99e08e95f1c872054968c3fc1ee1856f"
server_key_version: "encryptionkey@0"
I20260928 05:11:19.535416 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "511e83c9a31f4e64a55976254163f5c7"
format_stamp: "Formatted at 2026-09-28 05:11:19 on dist-test-slave-6718"
server_key: "e21a605ec2dd9bdd43b39a7712de8396"
server_key_iv: "99e08e95f1c872054968c3fc1ee1856f"
server_key_version: "encryptionkey@0"
I20260928 05:11:19.540335 6098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:11:19.543538 6113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:19.544929 6098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:11:19.545091 6098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "511e83c9a31f4e64a55976254163f5c7"
format_stamp: "Formatted at 2026-09-28 05:11:19 on dist-test-slave-6718"
server_key: "e21a605ec2dd9bdd43b39a7712de8396"
server_key_iv: "99e08e95f1c872054968c3fc1ee1856f"
server_key_version: "encryptionkey@0"
I20260928 05:11:19.545295 6098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:19.567467 6098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:19.571165 6098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:19.571451 6098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:19.572398 6098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:19.573697 6098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:19.573803 6098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:19.573954 6098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:19.574026 6098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:19.606298 6098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38835
I20260928 05:11:19.606352 6225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38835 every 8 connection(s)
I20260928 05:11:19.607556 6098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:19.615681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6098
I20260928 05:11:19.615896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:19.616488 388 external_mini_cluster.cc:1428] Setting key c8304a74e8f7b1f76999b05d38f4a9bc
Sep 28 05:11:19 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:19.628283 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:19.609966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39683 (local address 127.0.97.62:37195)
0928 05:11:19.610410 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:19.610416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:19.611403 (+ 987us) server_negotiation.cc:407] Connection header received
0928 05:11:19.613074 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:19.613078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:19.613150 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:19.613245 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:19.614679 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:19.615410 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:19.617195 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:19.617712 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:19.617815 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:19.621937 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:19.621946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:19.621953 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:19.621994 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:19.625066 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:19.625823 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:19.625831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:19.625838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:19.625977 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:19.626356 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:19.626364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:19.626370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:19.626764 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:19.626912 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:19.627567 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 05:11:19.627817 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":215,"threads_started":1}
I20260928 05:11:19.628932 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:19.610499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37195 (local address 127.0.97.1:39683)
0928 05:11:19.611245 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:19.611285 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:19.612832 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:19.613440 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:19.613447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:19.613942 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:19.614480 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:19.614496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:19.615559 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:19.615564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:19.617020 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:19.617035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:19.617941 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:19.617948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:19.618280 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:19.619052 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:19.619074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:19.621705 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:19.625201 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:19.625213 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:19.625233 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:19.625680 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:19.626078 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:19.626081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:19.626083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:19.626248 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:19.626875 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:19.626880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:19.627176 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 05:11:19.627409 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 05:11:19.627777 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":276,"threads_started":1}
I20260928 05:11:19.630476 6226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37195
I20260928 05:11:19.630798 6226 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:19.631557 6226 heartbeater.cc:511] Master 127.0.97.62:37195 requested a full tablet report, sending...
I20260928 05:11:19.633926 6037 ts_manager.cc:194] Registered new tserver with Master: 511e83c9a31f4e64a55976254163f5c7 (127.0.97.1:38835)
I20260928 05:11:19.636116 6037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39683
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:11:19.666392 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:19.982774 6232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:19.983072 6232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:19.983220 6232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:19.990370 6232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:19.990474 6232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:19.990670 6232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:20.000350 6232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:20.001588 6232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:20.002933 6232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:20.018831 6237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:20.018889 6240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:20.018868 6238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:20.019771 6232 server_base.cc:1034] running on GCE node
I20260928 05:11:20.020409 6232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:20.021413 6232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:20.022655 6232 hybrid_clock.cc:648] HybridClock initialized: now 1790572280022602 us; error 72 us; skew 500 ppm
I20260928 05:11:20.022974 6232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.025936 6232 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:11:20.027473 6232 webserver.cc:460] Webserver started at http://127.0.97.2:38577/ using document root <none> and password file <none>
I20260928 05:11:20.028160 6232 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:20.028302 6232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:20.028590 6232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:20.031564 6232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "723c9673bce445a1999c42a913702067"
format_stamp: "Formatted at 2026-09-28 05:11:20 on dist-test-slave-6718"
server_key: "53980f527a73dbab8960a2107db9f819"
server_key_iv: "38f654f00935970aae0169dfb86007c5"
server_key_version: "encryptionkey@0"
I20260928 05:11:20.032377 6232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "723c9673bce445a1999c42a913702067"
format_stamp: "Formatted at 2026-09-28 05:11:20 on dist-test-slave-6718"
server_key: "53980f527a73dbab8960a2107db9f819"
server_key_iv: "38f654f00935970aae0169dfb86007c5"
server_key_version: "encryptionkey@0"
I20260928 05:11:20.037395 6232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:11:20.040817 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:20.042296 6232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 05:11:20.042505 6232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "723c9673bce445a1999c42a913702067"
format_stamp: "Formatted at 2026-09-28 05:11:20 on dist-test-slave-6718"
server_key: "53980f527a73dbab8960a2107db9f819"
server_key_iv: "38f654f00935970aae0169dfb86007c5"
server_key_version: "encryptionkey@0"
I20260928 05:11:20.042789 6232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:20.065554 6232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:20.069828 6232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:20.070155 6232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:20.071308 6232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:20.072708 6232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:20.072821 6232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:20.072988 6232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:20.073086 6232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:20.110335 6232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33727
I20260928 05:11:20.110378 6359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33727 every 8 connection(s)
I20260928 05:11:20.112207 6232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:20.115658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6232
I20260928 05:11:20.115872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:20.116360 388 external_mini_cluster.cc:1428] Setting key 79b225785059f181a34a883a5793d233
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:20.134701 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.115315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57603 (local address 127.0.97.62:37195)
0928 05:11:20.117099 (+ 1784us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.117108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.117136 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:11:20.119585 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.119590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.119676 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.119811 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.121359 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.122308 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.124640 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.125124 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.125272 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.128621 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.128627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.128638 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.128668 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.131602 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.132424 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.132430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.132433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.132558 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.133009 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.133014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.133017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.133283 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.133428 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.134108 (+ 680us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.134332 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1640}
I20260928 05:11:20.135670 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.115937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37195 (local address 127.0.97.2:57603)
0928 05:11:20.116965 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:20.117043 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:20.119305 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:20.120064 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:20.120073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:20.120588 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:20.121184 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:20.121203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.122503 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.122511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:20.124461 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:20.124474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.125360 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.125367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:20.125595 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.126268 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:20.126294 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:20.128400 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:20.131732 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:20.131741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:20.131758 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:20.132281 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:20.132701 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:20.132706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:20.132709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:20.132891 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:20.133418 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:20.133426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:20.133757 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 05:11:20.134022 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 05:11:20.134561 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":448,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:11:20.137068 6360 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37195
I20260928 05:11:20.137392 6360 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:20.138072 6360 heartbeater.cc:511] Master 127.0.97.62:37195 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:11:20.139653 6037 ts_manager.cc:194] Registered new tserver with Master: 723c9673bce445a1999c42a913702067 (127.0.97.2:33727)
I20260928 05:11:20.140439 6037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:11:20.161768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:20.484651 6365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:20.484933 6365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:20.485097 6365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:20.492457 6365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:20.492588 6365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:20.492759 6365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:20.502457 6365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:20.503800 6365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:20.505093 6365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:20.517970 6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:20.518079 6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:20.518169 6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:20.518898 6365 server_base.cc:1034] running on GCE node
I20260928 05:11:20.519496 6365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:20.520421 6365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:20.521751 6365 hybrid_clock.cc:648] HybridClock initialized: now 1790572280521675 us; error 60 us; skew 500 ppm
I20260928 05:11:20.522130 6365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.525413 6365 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:11:20.526903 6365 webserver.cc:460] Webserver started at http://127.0.97.3:43361/ using document root <none> and password file <none>
I20260928 05:11:20.527560 6365 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:20.527701 6365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:20.527981 6365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:20.530901 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "8b044b18304d4f1dac078467488195a3"
format_stamp: "Formatted at 2026-09-28 05:11:20 on dist-test-slave-6718"
server_key: "a813750b2ea2ffa31ff5c92193830bc5"
server_key_iv: "245850e069f85bd58fea8b73074f2e01"
server_key_version: "encryptionkey@0"
I20260928 05:11:20.531714 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8b044b18304d4f1dac078467488195a3"
format_stamp: "Formatted at 2026-09-28 05:11:20 on dist-test-slave-6718"
server_key: "a813750b2ea2ffa31ff5c92193830bc5"
server_key_iv: "245850e069f85bd58fea8b73074f2e01"
server_key_version: "encryptionkey@0"
I20260928 05:11:20.536753 6365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 05:11:20.540207 6380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:20.541603 6365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:11:20.541785 6365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "8b044b18304d4f1dac078467488195a3"
format_stamp: "Formatted at 2026-09-28 05:11:20 on dist-test-slave-6718"
server_key: "a813750b2ea2ffa31ff5c92193830bc5"
server_key_iv: "245850e069f85bd58fea8b73074f2e01"
server_key_version: "encryptionkey@0"
I20260928 05:11:20.541981 6365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:20.580562 6365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:20.584980 6365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:20.585289 6365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:20.586268 6365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:20.587607 6365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:20.587714 6365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:20.587855 6365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:20.587925 6365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:20.619935 6365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35075
I20260928 05:11:20.619985 6492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35075 every 8 connection(s)
I20260928 05:11:20.621254 6365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:20.631018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6365
I20260928 05:11:20.631237 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:20.631753 388 external_mini_cluster.cc:1428] Setting key 82395f210488d58935dfe30bb9a921ef
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:20.639566 6226 heartbeater.cc:503] Master 127.0.97.62:37195 was elected leader, sending a full tablet report...
I20260928 05:11:20.640595 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.623870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52293 (local address 127.0.97.62:37195)
0928 05:11:20.624022 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.624026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.625155 (+ 1129us) server_negotiation.cc:407] Connection header received
0928 05:11:20.626956 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.626961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.627053 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.627161 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.628619 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.629359 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.630968 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.631482 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.631603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.635229 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.635234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.635239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.635280 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.637591 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.638241 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.638244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.638246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.638363 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.638771 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.638774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.638776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.639003 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.639108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.639979 (+ 871us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.640245 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 05:11:20.641602 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.624336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37195 (local address 127.0.97.3:52293)
0928 05:11:20.625001 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:20.625036 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:20.626710 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:20.627391 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:20.627399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:20.627871 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:20.628462 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:20.628479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.629497 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.629504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:20.630798 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:20.630810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.631696 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.631705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:20.631987 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.632646 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:20.632675 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:20.635045 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:20.637720 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:20.637726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:20.637737 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:20.638136 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:20.638451 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:20.638454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:20.638456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:20.638672 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:20.639108 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:20.639114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:20.639463 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 05:11:20.639754 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 05:11:20.640233 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
I20260928 05:11:20.642988 6493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37195
I20260928 05:11:20.643313 6493 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:20.644058 6493 heartbeater.cc:511] Master 127.0.97.62:37195 requested a full tablet report, sending...
I20260928 05:11:20.645689 6037 ts_manager.cc:194] Registered new tserver with Master: 8b044b18304d4f1dac078467488195a3 (127.0.97.3:35075)
I20260928 05:11:20.646713 6037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52293
I20260928 05:11:20.647154 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/test-admin.keytab.
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.673841 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 05:11:20.678725 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:20.676290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.97.1:38835)
0928 05:11:20.676915 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.676924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.676956 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:11:20.677036 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.677042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.677261 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.677430 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.678278 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52570: BlockingRecv error: recv got EOF from 127.0.0.1:52570 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":289,"threads_started":1}
W20260928 05:11:20.678942 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.680469 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.681808 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:20.696503 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.683334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.0.97.1:38835)
0928 05:11:20.683532 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.683539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.683820 (+ 281us) server_negotiation.cc:407] Connection header received
0928 05:11:20.683995 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.684000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.684065 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.684175 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.685079 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.686030 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.687344 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.687912 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.688053 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.691491 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.691502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.691516 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.691547 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.694098 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.694814 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.694821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.694823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.694947 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.695255 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.695258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.695259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.695577 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.695783 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.696053 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.696244 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 05:11:20.702275 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:20.700405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.97.1:38835)
0928 05:11:20.700621 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.700629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.700877 (+ 248us) server_negotiation.cc:407] Connection header received
0928 05:11:20.701062 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.701067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.701131 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.701247 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.702022 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52588: BlockingRecv error: recv got EOF from 127.0.0.1:52588 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 05:11:20.702442 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.703971 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.705237 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:20.718724 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.706546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.0.97.1:38835)
0928 05:11:20.706751 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.706758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.706969 (+ 211us) server_negotiation.cc:407] Connection header received
0928 05:11:20.707156 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.707161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.707232 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.707333 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.708295 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.708985 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.710206 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.710686 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.710835 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.714140 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.714144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.714148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.714190 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.716593 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.717143 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.717147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.717149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.717257 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.717553 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.717556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.717558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.717800 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.717986 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.718220 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.718362 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:11:20.722920 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:20.721123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.97.1:38835)
0928 05:11:20.721279 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.721283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.721662 (+ 379us) server_negotiation.cc:407] Connection header received
0928 05:11:20.721774 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.721777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.721840 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.721915 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.722669 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52600: BlockingRecv error: recv got EOF from 127.0.0.1:52600 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 05:11:20.723086 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790572228671169-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.724525 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:20.725757 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:20.739184 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.727113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.97.1:38835)
0928 05:11:20.727296 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.727309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.727411 (+ 102us) server_negotiation.cc:407] Connection header received
0928 05:11:20.727526 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.727530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.727582 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.727650 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.728477 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.729169 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.730305 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.730796 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.730929 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.734237 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.734241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.734244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.734273 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.736582 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.737251 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.737257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.737262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.737390 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.737748 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.737755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.737786 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.738070 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.738380 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.738671 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.738881 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:11:20.740854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6098
I20260928 05:11:20.753293 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.741598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.0.97.1:38835)
0928 05:11:20.741786 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.741794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.742092 (+ 298us) server_negotiation.cc:407] Connection header received
0928 05:11:20.742231 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.742234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.742289 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.742366 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.743313 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.743963 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.745444 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.745899 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.745997 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.748343 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.748348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.748351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.748384 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.751038 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.751629 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.751634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.751636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.751762 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.752064 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.752067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.752069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.752323 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.752511 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.752751 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.752915 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:11:20.753852 6221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:20.836107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6098
I20260928 05:11:20.852471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6232
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:11:20.867089 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.853400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.97.2:33727)
0928 05:11:20.853920 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.853927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.853953 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:11:20.854009 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.854015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.854233 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.854361 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.855296 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.856285 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.857458 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.858058 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.858182 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.861760 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.861771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.861786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.861824 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.864649 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.865241 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.865248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.865250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.865364 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.865665 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.865668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.865670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.866000 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.866218 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.866455 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.866761 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":295,"threads_started":1}
I20260928 05:11:20.868302 6355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:20.954276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6232
I20260928 05:11:20.970085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6365
Sep 28 05:11:20 dist-test-slave-6718 krb5kdc[5992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:11:20.986197 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:20.970973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.0.97.3:35075)
0928 05:11:20.971533 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:20.971540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:20.971567 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:11:20.971625 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:20.971631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:20.971847 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:20.971976 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:20.972932 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.973931 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.975191 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:20.975726 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:20.975872 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:20.980051 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:20.980061 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:20.980077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:20.980112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:20.983464 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.984112 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.984120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.984124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.984263 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:20.984584 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:20.984588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:20.984592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:20.985007 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:20.985321 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:20.985542 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 05:11:20.985812 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":282,"threads_started":1}
I20260928 05:11:20.987612 6488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:21.069299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6365
I20260928 05:11:21.085021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6005
I20260928 05:11:21.086225 6067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:21.187518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6005
2026-09-28T05:11:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:21 dist-test-slave-6718 krb5kdc[6520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:21 dist-test-slave-6718 krb5kdc[6520](info): set up 2 sockets
Sep 28 05:11:21 dist-test-slave-6718 krb5kdc[6520](info): commencing operation
krb5kdc: starting...
W20260928 05:11:22.275274 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:22 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 05:11:22.488174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43605
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:22.808619 6533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:22.808898 6533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:22.809062 6533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:22.816105 6533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:22.816212 6533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:22.816286 6533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:22.816367 6533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:22.816442 6533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:22.825927 6533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:22.827327 6533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:22.828662 6533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:22.840147 6538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:22.840207 6541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:22.840188 6539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:22.840977 6533 server_base.cc:1034] running on GCE node
I20260928 05:11:22.841812 6533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:22.843309 6533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:22.844540 6533 hybrid_clock.cc:648] HybridClock initialized: now 1790572282844498 us; error 55 us; skew 500 ppm
I20260928 05:11:22.844898 6533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:22 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:22.848052 6533 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 05:11:22.849452 6533 webserver.cc:460] Webserver started at http://127.0.97.62:36597/ using document root <none> and password file <none>
I20260928 05:11:22.850128 6533 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:22.850277 6533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:22.850626 6533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:22.853490 6533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "540972e76935476f91b35f900d41dae7"
format_stamp: "Formatted at 2026-09-28 05:11:22 on dist-test-slave-6718"
server_key: "0b215db0edbf7c5a9047801adf319cdc"
server_key_iv: "fd42dc43cf2608f1976132cf9d9721ee"
server_key_version: "encryptionkey@0"
I20260928 05:11:22.854255 6533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "540972e76935476f91b35f900d41dae7"
format_stamp: "Formatted at 2026-09-28 05:11:22 on dist-test-slave-6718"
server_key: "0b215db0edbf7c5a9047801adf319cdc"
server_key_iv: "fd42dc43cf2608f1976132cf9d9721ee"
server_key_version: "encryptionkey@0"
I20260928 05:11:22.859253 6533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 05:11:22.862430 6548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:22.863780 6533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:11:22.863971 6533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "540972e76935476f91b35f900d41dae7"
format_stamp: "Formatted at 2026-09-28 05:11:22 on dist-test-slave-6718"
server_key: "0b215db0edbf7c5a9047801adf319cdc"
server_key_iv: "fd42dc43cf2608f1976132cf9d9721ee"
server_key_version: "encryptionkey@0"
I20260928 05:11:22.864166 6533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:22.883744 6533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:22.887868 6533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:22.888142 6533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:22.905880 6533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43605
I20260928 05:11:22.905915 6599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43605 every 8 connection(s)
I20260928 05:11:22.907258 6533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:22.908357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6533
I20260928 05:11:22.908581 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:22.909324 388 external_mini_cluster.cc:1428] Setting key 210b779ac7955670ba6daa30f51bb6f6
I20260928 05:11:22.911887 6600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:11:22 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:22.924502 6600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Bootstrap starting.
I20260928 05:11:22.928663 6600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:22.929966 6600 log.cc:824] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:22.933144 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:22.911294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.0.97.62:43605)
0928 05:11:22.912098 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:22.912110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:22.912149 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:11:22.913875 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:22.913897 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:22.914294 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:22.914704 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:22.916370 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:22.917655 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:22.919367 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:22.919993 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:22.920116 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:22.924560 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:22.924573 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:22.924593 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:22.924641 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:22.928492 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:22.929247 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:22.929255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:22.929281 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:22.929475 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:22.929831 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:22.929836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:22.929839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:22.930594 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:22.930870 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:22.931347 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 05:11:22.931814 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":372,"threads_started":1}
I20260928 05:11:22.933497 6600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: No bootstrap required, opened a new log
I20260928 05:11:22.941561 6600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } }
I20260928 05:11:22.941901 6600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:22.941978 6600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540972e76935476f91b35f900d41dae7, State: Initialized, Role: FOLLOWER
I20260928 05:11:22.942456 6600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } }
I20260928 05:11:22.942711 6600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:22.942813 6600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:22.942938 6600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:22.944948 6600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } }
I20260928 05:11:22.945497 6600 leader_election.cc:302] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540972e76935476f91b35f900d41dae7; no voters:
I20260928 05:11:22.946153 6600 leader_election.cc:288] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:22.946314 6605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:22.947033 6605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 LEADER]: Becoming Leader. State: Replica: 540972e76935476f91b35f900d41dae7, State: Running, Role: LEADER
I20260928 05:11:22.947525 6605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } }
I20260928 05:11:22.948207 6600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:22.950387 6607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540972e76935476f91b35f900d41dae7. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } }
I20260928 05:11:22.951494 6607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:22.950822 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } }
I20260928 05:11:22.951810 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:22.954000 6612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:22.960182 6612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:22.969772 6612 catalog_manager.cc:1347] Generated new cluster ID: 3a2752c27580474ca021d7bf60aa5459
I20260928 05:11:22.969889 6612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:22.992271 6612 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:22.994172 6612 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:23.019672 6612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Generated new TSK 0
I20260928 05:11:23.020709 6612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 05:11:23.069573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43605
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:23.391021 6626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:23.391455 6626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:23.391592 6626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:23.398739 6626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:23.398840 6626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:23.398948 6626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:23.408547 6626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:23.409788 6626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:23.411046 6626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:23.423242 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:23.423171 6634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:23.423207 6632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:23.424140 6626 server_base.cc:1034] running on GCE node
I20260928 05:11:23.424710 6626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:23.425590 6626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:23.426819 6626 hybrid_clock.cc:648] HybridClock initialized: now 1790572283426783 us; error 54 us; skew 500 ppm
I20260928 05:11:23.427163 6626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:23 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:23.430269 6626 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:11:23.431859 6626 webserver.cc:460] Webserver started at http://127.0.97.1:44321/ using document root <none> and password file <none>
I20260928 05:11:23.432566 6626 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:23.432664 6626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:23.432896 6626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:23.435808 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "5e0c6f7dbe174d28976b26e4ba2be73a"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "5b5985f037183486ccfc25b7b15ad3c0"
server_key_iv: "5096e8ce4186bac6fd5948b06f748467"
server_key_version: "encryptionkey@0"
I20260928 05:11:23.436553 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e0c6f7dbe174d28976b26e4ba2be73a"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "5b5985f037183486ccfc25b7b15ad3c0"
server_key_iv: "5096e8ce4186bac6fd5948b06f748467"
server_key_version: "encryptionkey@0"
I20260928 05:11:23.441540 6626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 05:11:23.444787 6641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:23.446025 6626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:11:23.446223 6626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "5e0c6f7dbe174d28976b26e4ba2be73a"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "5b5985f037183486ccfc25b7b15ad3c0"
server_key_iv: "5096e8ce4186bac6fd5948b06f748467"
server_key_version: "encryptionkey@0"
I20260928 05:11:23.446414 6626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:23.471366 6626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:23.475649 6626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:23.475952 6626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:23.476866 6626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:23.478132 6626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:23.478201 6626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:23.478366 6626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:23.478426 6626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:23.508399 6626 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45689
I20260928 05:11:23.508440 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45689 every 8 connection(s)
I20260928 05:11:23.509661 6626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:23.519136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6626
I20260928 05:11:23.519369 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:23.520083 388 external_mini_cluster.cc:1428] Setting key 7173afda1d321eace6d60f9d9b70f9ea
Sep 28 05:11:23 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:23.529135 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:23.512111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55829 (local address 127.0.97.62:43605)
0928 05:11:23.512557 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:23.512561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:23.513294 (+ 733us) server_negotiation.cc:407] Connection header received
0928 05:11:23.515118 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:23.515124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:23.515221 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:23.515335 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:23.516708 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:23.517424 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:23.519261 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:23.519802 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:23.519981 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:23.523498 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:23.523503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:23.523506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:23.523535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:23.526057 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:23.526846 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:23.526854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:23.526857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:23.527005 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:23.527477 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:23.527482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:23.527485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:23.527749 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:23.527904 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:23.528575 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 05:11:23.528792 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
I20260928 05:11:23.530056 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:23.512486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:55829)
0928 05:11:23.513144 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:23.513181 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:23.514871 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:23.515553 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:23.515560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:23.516033 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:23.516546 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:23.516561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:23.517570 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:23.517580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:23.519048 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:23.519064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:23.520049 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:23.520057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:23.520350 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:23.521009 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:23.521031 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:23.523280 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:23.526197 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:23.526206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:23.526230 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:23.526723 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:23.527147 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:23.527153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:23.527156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:23.527359 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:23.527882 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:23.527891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:23.528235 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 05:11:23.528489 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 05:11:23.528987 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":256,"threads_started":1}
I20260928 05:11:23.531668 6754 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43605
I20260928 05:11:23.532001 6754 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:23.532831 6754 heartbeater.cc:511] Master 127.0.97.62:43605 requested a full tablet report, sending...
I20260928 05:11:23.534997 6565 ts_manager.cc:194] Registered new tserver with Master: 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689)
I20260928 05:11:23.536355 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55829
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 05:11:23.567517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43605
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:23.889011 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:23.889292 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:23.889384 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:23.896466 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:23.896576 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:23.896687 6760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:23.906307 6760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:23.907541 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:23.908798 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:23.923337 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:23.923367 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:23.923615 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:23.924332 6760 server_base.cc:1034] running on GCE node
I20260928 05:11:23.924880 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:23.925776 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:23.927008 6760 hybrid_clock.cc:648] HybridClock initialized: now 1790572283926950 us; error 54 us; skew 500 ppm
I20260928 05:11:23.927335 6760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:23 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:23.930387 6760 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:11:23.931787 6760 webserver.cc:460] Webserver started at http://127.0.97.2:43385/ using document root <none> and password file <none>
I20260928 05:11:23.932480 6760 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:23.932626 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:23.932924 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:23.935930 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "d441a081acef4a40800dd49ad09f5f29"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "d1825cca9006d7c50c53b8940c3bdd18"
server_key_iv: "3ca73089c4a8a3c59143421f55be4791"
server_key_version: "encryptionkey@0"
I20260928 05:11:23.936725 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d441a081acef4a40800dd49ad09f5f29"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "d1825cca9006d7c50c53b8940c3bdd18"
server_key_iv: "3ca73089c4a8a3c59143421f55be4791"
server_key_version: "encryptionkey@0"
I20260928 05:11:23.941867 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:11:23.945343 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:23.946753 6760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:23.946919 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "d441a081acef4a40800dd49ad09f5f29"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "d1825cca9006d7c50c53b8940c3bdd18"
server_key_iv: "3ca73089c4a8a3c59143421f55be4791"
server_key_version: "encryptionkey@0"
I20260928 05:11:23.947121 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:23.969704 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:23.973796 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:23.974098 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:23.975111 6760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:23.976428 6760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:23.976536 6760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:23.976686 6760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:23.976758 6760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:24.010152 6760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38161
I20260928 05:11:24.010248 6887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38161 every 8 connection(s)
I20260928 05:11:24.011536 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:24.017902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6760
I20260928 05:11:24.018122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:24.018654 388 external_mini_cluster.cc:1428] Setting key fba876e0ba2cfdef267992be2611f732
Sep 28 05:11:24 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:24.035305 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:24.014016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56371 (local address 127.0.97.62:43605)
0928 05:11:24.014192 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:24.014196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:24.015380 (+ 1184us) server_negotiation.cc:407] Connection header received
0928 05:11:24.017130 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:24.017135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:24.017227 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:24.017336 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:24.019386 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.020514 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.022993 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.023390 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.023494 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:24.028026 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:24.028034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:24.028041 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:24.028088 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:24.031787 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:24.032566 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:24.032574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:24.032581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:24.032738 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:24.033255 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:24.033263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:24.033269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:24.033641 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:24.033790 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:24.034590 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 05:11:24.034857 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:11:24.036044 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:24.014485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:56371)
0928 05:11:24.015233 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:24.015269 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:24.016879 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:24.017624 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:24.017634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:24.018405 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:24.019204 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:24.019235 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.020716 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.020721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:24.022841 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:24.022858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.023618 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.023627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:24.023959 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:24.024669 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:24.024702 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:24.027739 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:24.031958 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:24.031965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:24.031981 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:24.032419 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:24.032890 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:24.032894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:24.032897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:24.033101 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:24.033790 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:24.033796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:24.034157 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 05:11:24.034400 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 05:11:24.034891 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":257,"threads_started":1}
I20260928 05:11:24.037669 6888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43605
I20260928 05:11:24.037981 6888 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:24.038781 6888 heartbeater.cc:511] Master 127.0.97.62:43605 requested a full tablet report, sending...
I20260928 05:11:24.040261 6565 ts_manager.cc:194] Registered new tserver with Master: d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161)
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:11:24.041052 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 05:11:24.067457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43605
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:24.392640 6893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:24.392920 6893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:24.393107 6893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:24.400260 6893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:24.400359 6893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:24.400557 6893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:24.410043 6893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:24.411310 6893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:24.412737 6893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:24.425434 6899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:24.425398 6901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:24.425467 6898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:24.426163 6893 server_base.cc:1034] running on GCE node
I20260928 05:11:24.426867 6893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:24.427822 6893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:24.429145 6893 hybrid_clock.cc:648] HybridClock initialized: now 1790572284429103 us; error 58 us; skew 500 ppm
I20260928 05:11:24.429463 6893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:24 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:24.432546 6893 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:11:24.433964 6893 webserver.cc:460] Webserver started at http://127.0.97.3:45127/ using document root <none> and password file <none>
I20260928 05:11:24.434706 6893 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:24.434851 6893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:24.435142 6893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:24.438068 6893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "1b03c854497442c99bc3a3dfcfaa3534"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "b1b2f880c1956d91a587f340f2ab5810"
server_key_iv: "9b222fd02cfc5cfc00d287af888526c9"
server_key_version: "encryptionkey@0"
I20260928 05:11:24.438915 6893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1b03c854497442c99bc3a3dfcfaa3534"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "b1b2f880c1956d91a587f340f2ab5810"
server_key_iv: "9b222fd02cfc5cfc00d287af888526c9"
server_key_version: "encryptionkey@0"
I20260928 05:11:24.443948 6893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 05:11:24.447674 6908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:24.449072 6893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:11:24.449239 6893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "1b03c854497442c99bc3a3dfcfaa3534"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "b1b2f880c1956d91a587f340f2ab5810"
server_key_iv: "9b222fd02cfc5cfc00d287af888526c9"
server_key_version: "encryptionkey@0"
I20260928 05:11:24.449434 6893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:24.480237 6893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:24.484282 6893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:24.484576 6893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:24.485507 6893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:24.486860 6893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:24.486958 6893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:24.487097 6893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:24.487177 6893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:24.520053 6893 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37397
I20260928 05:11:24.520102 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37397 every 8 connection(s)
I20260928 05:11:24.521283 6893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:24.528254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 6893
I20260928 05:11:24.528455 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:24.528869 388 external_mini_cluster.cc:1428] Setting key 9b98d2aaebbf47bb8fadd96ad881723a
Sep 28 05:11:24 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:24.540124 6754 heartbeater.cc:503] Master 127.0.97.62:43605 was elected leader, sending a full tablet report...
I20260928 05:11:24.544517 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:24.523745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34799 (local address 127.0.97.62:43605)
0928 05:11:24.523921 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:24.523926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:24.525014 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 05:11:24.526824 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:24.526829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:24.526917 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:24.527026 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:24.528411 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.529254 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.531143 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.531662 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.531789 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:24.538386 (+ 6597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:24.538391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:24.538395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:24.538428 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:24.540996 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:24.541914 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:24.541919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:24.541921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:24.542045 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:24.542521 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:24.542576 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:24.542579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:24.542942 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:24.543090 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:24.544098 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0928 05:11:24.544278 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:11:24.545531 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:24.524199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:34799)
0928 05:11:24.524856 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:24.524896 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:24.526506 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:24.527253 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:24.527260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:24.527726 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:24.528247 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:24.528262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.529390 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.529397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:24.531004 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:24.531015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:24.531886 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:24.531895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:24.532132 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:24.535996 (+ 3864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:24.536018 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:24.538170 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:24.541150 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:24.541159 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:24.541176 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:24.541786 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:24.542162 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:24.542168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:24.542171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:24.542397 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:24.543133 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:24.543142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:24.543603 (+ 461us) client_negotiation.cc:769] Sending connection context
0928 05:11:24.543936 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 05:11:24.544404 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":263,"threads_started":1}
I20260928 05:11:24.546993 7021 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43605
I20260928 05:11:24.547302 7021 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:24.548100 7021 heartbeater.cc:511] Master 127.0.97.62:43605 requested a full tablet report, sending...
I20260928 05:11:24.549546 6565 ts_manager.cc:194] Registered new tserver with Master: 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397)
I20260928 05:11:24.550403 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34799
I20260928 05:11:24.556059 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260928 05:11:24.602996 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34975
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43605,127.0.97.61:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:24.922492 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:24.922868 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:24.923009 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:24.930078 7026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:24.930214 7026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:24.930313 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:24.930406 7026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:24.930476 7026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:24.940282 7026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43605,127.0.97.61:34975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
W20260928 05:11:24.940692 7026 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:43605,127.0.97.61:34975'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 05:11:24.941763 7026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:24.943140 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:24.955075 7032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:24.955309 7034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:24.955686 7031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:24.955964 7026 server_base.cc:1034] running on GCE node
I20260928 05:11:24.956743 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:24.958266 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:24.959470 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790572284959390 us; error 70 us; skew 500 ppm
I20260928 05:11:24.959794 7026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:24 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:24.962961 7026 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 05:11:24.964533 7026 webserver.cc:460] Webserver started at http://127.0.97.61:42447/ using document root <none> and password file <none>
I20260928 05:11:24.965243 7026 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:24.965368 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:24.965672 7026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:24.968643 7026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/instance:
uuid: "50b19a82457942dc91b0ee156ee92f21"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "3a7ebd55de6930f6adf2a3ccf8ae799f"
server_key_iv: "6d569bc4a322fa1ed4d4413c280967c0"
server_key_version: "encryptionkey@0"
I20260928 05:11:24.969450 7026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal/instance:
uuid: "50b19a82457942dc91b0ee156ee92f21"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "3a7ebd55de6930f6adf2a3ccf8ae799f"
server_key_iv: "6d569bc4a322fa1ed4d4413c280967c0"
server_key_version: "encryptionkey@0"
I20260928 05:11:24.974812 7026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 05:11:24.978214 7041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:24.979595 7026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:24.979785 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
uuid: "50b19a82457942dc91b0ee156ee92f21"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "3a7ebd55de6930f6adf2a3ccf8ae799f"
server_key_iv: "6d569bc4a322fa1ed4d4413c280967c0"
server_key_version: "encryptionkey@0"
I20260928 05:11:24.980001 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:25.001806 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:25.006321 7026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:25.006644 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:25.026309 7026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34975
I20260928 05:11:25.026328 7092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34975 every 8 connection(s)
I20260928 05:11:25.027740 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:11:25.031374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 7026
I20260928 05:11:25.031697 7093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:25.041200 7093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:11:25.043877 6888 heartbeater.cc:503] Master 127.0.97.62:43605 was elected leader, sending a full tablet report...
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:25.056015 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.033590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.0.97.61:34975)
0928 05:11:25.034342 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.034361 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.034413 (+ 52us) server_negotiation.cc:407] Connection header received
0928 05:11:25.036546 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.036574 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.037123 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.037578 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.039073 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.040380 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.041673 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.042408 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.042627 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.047689 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.047697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.047707 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.047741 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.051833 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.052636 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.052643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.052649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.052809 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.053226 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.053235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.053240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.053799 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.054089 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.054591 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.055031 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260928 05:11:25.059549 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.043787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.0.97.62:43605)
0928 05:11:25.043983 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.043991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.044843 (+ 852us) server_negotiation.cc:407] Connection header received
0928 05:11:25.045125 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.045130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.045198 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.045297 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.046644 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.047411 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.049094 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.049806 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.049957 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.053921 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.053928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.053933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.053970 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.056711 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.057415 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.057420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.057424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.057571 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.057936 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.057943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.057946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.058305 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.058456 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.059010 (+ 554us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.059216 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:11:25.059599 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.043806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:36196)
0928 05:11:25.044715 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.044756 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.044954 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.045362 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.045368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.046085 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.046432 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.046447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.047551 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.047559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.048957 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.048969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.050223 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.050229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.050418 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.051111 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.051136 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.053737 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.056842 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.056853 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.056860 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.057293 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.057687 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.057693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.057697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.057835 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.058442 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.058450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.058689 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.058915 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.059213 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":520,"threads_started":1}
I20260928 05:11:25.060988 7093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572284, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:11:25.072563 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.061733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:51750)
0928 05:11:25.061885 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.061900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.061976 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.062275 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.062277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.062642 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.062776 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.062783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.063706 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.063709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.064521 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.064530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.065131 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.065135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.065274 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.065747 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.065762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.067690 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.070442 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.070446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.070448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.070911 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.071304 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.071307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.071309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.071424 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.071862 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.071864 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.071972 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.072076 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.072283 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:11:25.072729 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.061900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.0.97.61:34975)
0928 05:11:25.062034 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.062038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.062051 (+ 13us) server_negotiation.cc:407] Connection header received
0928 05:11:25.062086 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.062089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.062173 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.062248 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.062908 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.063589 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.064633 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.065009 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.065115 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.067845 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.067850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.067853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.067880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.070318 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.071055 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.071092 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.071095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.071211 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.071513 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.071516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.071518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.071774 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.071889 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.072181 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.072401 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":45}
I20260928 05:11:25.077256 7093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Bootstrap starting.
I20260928 05:11:25.080302 7093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:25.081235 7093 log.cc:824] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:25.083942 7093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: No bootstrap required, opened a new log
I20260928 05:11:25.090837 7093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } }
I20260928 05:11:25.091137 7093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:25.091253 7093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b19a82457942dc91b0ee156ee92f21, State: Initialized, Role: FOLLOWER
I20260928 05:11:25.091807 7093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } }
I20260928 05:11:25.092639 7099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } } }
I20260928 05:11:25.092976 7099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:25.093616 7093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:11:25.102442 7110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:11:25.102608 7110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:11:25.104751 7026 master_runner.cc:428] Detected that this master 50b19a82457942dc91b0ee156ee92f21 is joining an existing cluster
I20260928 05:11:25.104847 7026 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:34975
I20260928 05:11:25.124441 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.108791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.0.97.62:43605)
0928 05:11:25.108970 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.108976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.109660 (+ 684us) server_negotiation.cc:407] Connection header received
0928 05:11:25.109964 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.109970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.110047 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.110112 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.111571 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.112630 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.114676 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.115310 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.115404 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.117831 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.117838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.117843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.117880 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.121174 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.122029 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.122035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.122038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.122189 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.122678 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.122684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.122687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.123033 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.123163 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.123838 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.124071 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:11:25.125602 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.108915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:36204)
0928 05:11:25.109553 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.109572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.109807 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.110339 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.110343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.111004 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.111405 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.111417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.112812 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.112817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.114500 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.114514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.115682 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.115688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.115875 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.116526 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.116548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.117622 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.121340 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.121345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.121349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.121884 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.122359 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.122363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.122367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.122551 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.123279 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.123283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.123493 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.124798 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.125117 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":373,"threads_started":1}
I20260928 05:11:25.136656 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.108915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.0.97.61:34975)
0928 05:11:25.109408 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.109417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.120587 (+ 11170us) server_negotiation.cc:407] Connection header received
0928 05:11:25.120816 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.120823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.120894 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.120960 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.122452 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.123228 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.125605 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.126472 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.126614 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.129413 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.129422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.129430 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.129479 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.133326 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.134210 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.134219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.134224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.134382 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.134879 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.134886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.134890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.135327 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.135483 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.135999 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.136242 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260928 05:11:25.138036 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.109985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:51760)
0928 05:11:25.120489 (+ 10504us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.120511 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.120687 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.121163 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.121169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.121928 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.122328 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.122344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.123402 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.123413 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.125433 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.125449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.126909 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.126917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.127132 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.127975 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.128004 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.129194 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.133485 (+ 4291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.133495 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.133502 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.134076 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.134560 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.134564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.134569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.134739 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.135565 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.135570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.135725 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.137223 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.137556 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10152,"thread_start_us":9844,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:11:25.159022 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.132848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.0.97.61:34975)
0928 05:11:25.137183 (+ 4335us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.137190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.139717 (+ 2527us) server_negotiation.cc:407] Connection header received
0928 05:11:25.140091 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.140102 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.140184 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.140298 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.142127 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.143314 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.145338 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.145997 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.146122 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.150786 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.150795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.150803 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.150848 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.155070 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.156020 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.156031 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.156038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.156206 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.156627 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.156634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.156640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.157417 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.157578 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.158281 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.158513 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4215,"thread_start_us":5998,"threads_started":1}
I20260928 05:11:25.159376 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.132467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:51772)
0928 05:11:25.139588 (+ 7121us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.139632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.139858 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.140345 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.140350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.141652 (+ 1302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.141955 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.141977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.143462 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.143469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.145188 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.145205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.146170 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.146179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.146415 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.147226 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.147255 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.150469 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.155281 (+ 4812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.155295 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.155303 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.155886 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.156325 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.156330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.156334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.156507 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.157661 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.157673 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.157902 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.158156 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.158615 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6858,"thread_start_us":6675,"threads_started":1}
I20260928 05:11:25.161449 6565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:34975: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50b19a82457942dc91b0ee156ee92f21"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 34975
}
attrs {
promote: true
}
}
dest_uuid: "540972e76935476f91b35f900d41dae7"
cas_config_opid_index: -1
I20260928 05:11:25.162588 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } }
I20260928 05:11:25.165807 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } }
I20260928 05:11:25.166826 7068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:25.169510 7068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Refusing update from remote peer 540972e76935476f91b35f900d41dae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 05:11:25.170069 7124 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:34975
I20260928 05:11:25.170554 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } }
I20260928 05:11:25.170876 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:25.170960 7026 master.cc:537] Master@127.0.97.61:34975 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260928 05:11:25.173884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45951
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:11:25.176328 7123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:25.180079 7099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:25.180361 7099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } }
I20260928 05:11:25.181866 7099 leader_election.cc:288] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605)
I20260928 05:11:25.182768 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50b19a82457942dc91b0ee156ee92f21" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540972e76935476f91b35f900d41dae7" is_pre_election: true
I20260928 05:11:25.184063 7042 leader_election.cc:302] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 50b19a82457942dc91b0ee156ee92f21; no voters: 540972e76935476f91b35f900d41dae7
I20260928 05:11:25.184754 7099 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 05:11:25.184988 6552 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 05:11:25.186728 6552 consensus_peers.cc:487] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 -> Peer 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61:34975): Couldn't send request to peer 50b19a82457942dc91b0ee156ee92f21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 05:11:25.189730 7026 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:25.190191 7026 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:25.190466 7026 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: stopping tablet replica
I20260928 05:11:25.197324 7026 master.cc:552] Master@127.0.97.61:34975 shutdown complete.
I20260928 05:11:25.197476 7026 master_runner.cc:305] Clearing existing system tablet
I20260928 05:11:25.197965 7026 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:25.198130 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:25.201139 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 05:11:25.202919 7130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:25.203246 7026 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:25.203667 7026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 05:11:25.203794 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
uuid: "50b19a82457942dc91b0ee156ee92f21"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "3a7ebd55de6930f6adf2a3ccf8ae799f"
server_key_iv: "6d569bc4a322fa1ed4d4413c280967c0"
server_key_version: "encryptionkey@0"
I20260928 05:11:25.208730 7026 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:25.215726 7026 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 05:11:25.215993 7026 log.cc:1196] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 05:11:25.216508 7026 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Deleting consensus metadata
I20260928 05:11:25.216840 7026 master_runner.cc:315] Copying system tablet from 127.0.97.62:43605
I20260928 05:11:25.219328 7026 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43605
I20260928 05:11:25.237413 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.220164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.0.97.62:43605)
0928 05:11:25.220391 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.220397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.223059 (+ 2662us) server_negotiation.cc:407] Connection header received
0928 05:11:25.223253 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.223259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.223354 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.223469 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.224471 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.225610 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.227433 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.228157 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.228321 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.230479 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.230485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.230490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.230551 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.234296 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.235096 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.235103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.235109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.235290 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.235729 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.235736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.235741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.236221 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.236394 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.236832 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.237088 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:11:25.237476 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.220131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:36208)
0928 05:11:25.222919 (+ 2788us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.222940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.223116 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.223505 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.223510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.224025 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.224286 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.224298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.225769 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.225775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.227283 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.227298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.228326 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.228332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.228523 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.229190 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.229221 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.230275 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.234446 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.234452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.234456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.234967 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.235418 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.235423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.235427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.235585 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.236348 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.236353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.236533 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.236678 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.236996 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2633,"thread_start_us":2538,"threads_started":1}
I20260928 05:11:25.238584 6585 tablet_copy_service.cc:140] P 540972e76935476f91b35f900d41dae7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50b19a82457942dc91b0ee156ee92f21 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36208)
I20260928 05:11:25.238837 6585 tablet_copy_service.cc:161] P 540972e76935476f91b35f900d41dae7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50b19a82457942dc91b0ee156ee92f21 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36208: session id = 50b19a82457942dc91b0ee156ee92f21-00000000000000000000000000000000
I20260928 05:11:25.243188 6585 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 05:11:25.245461 7026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 05:11:25.252769 7026 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: tablet copy: Starting download of 0 data blocks...
I20260928 05:11:25.253082 7026 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: tablet copy: Starting download of 1 WAL segments...
I20260928 05:11:25.255935 7026 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 05:11:25.260591 6585 tablet_copy_service.cc:342] P 540972e76935476f91b35f900d41dae7: Request end of tablet copy session 50b19a82457942dc91b0ee156ee92f21-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:36208
I20260928 05:11:25.260788 6585 tablet_copy_service.cc:433] P 540972e76935476f91b35f900d41dae7: ending tablet copy session 50b19a82457942dc91b0ee156ee92f21-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50b19a82457942dc91b0ee156ee92f21
I20260928 05:11:25.273293 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:25.275872 7137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:25.280043 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:25.288002 7026 server_base.cc:1034] running on GCE node
W20260928 05:11:25.290792 7140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:25.291317 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:25.294786 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:25.296110 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790572285296044 us; error 37 us; skew 500 ppm
I20260928 05:11:25.296365 7026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572285, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:25.298633 7026 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 05:11:25.299971 7026 webserver.cc:460] Webserver started at http://127.0.97.61:38051/ using document root <none> and password file <none>
I20260928 05:11:25.300359 7026 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:25.300450 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:25.303555 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 05:11:25.305471 7147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:25.305899 7026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 05:11:25.306023 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
uuid: "50b19a82457942dc91b0ee156ee92f21"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "3a7ebd55de6930f6adf2a3ccf8ae799f"
server_key_iv: "6d569bc4a322fa1ed4d4413c280967c0"
server_key_version: "encryptionkey@0"
I20260928 05:11:25.306196 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:25.336536 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:25.341224 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:25.362995 7026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34975
I20260928 05:11:25.363035 7198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34975 every 8 connection(s)
I20260928 05:11:25.364023 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:11:25.369428 7199 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:11:25.373697 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Bootstrap starting.
I20260928 05:11:25.391251 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:11:25.391808 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: Bootstrap complete.
I20260928 05:11:25.393077 7199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } }
I20260928 05:11:25.393401 7199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50b19a82457942dc91b0ee156ee92f21, State: Initialized, Role: LEARNER
I20260928 05:11:25.393688 7199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } }
I20260928 05:11:25.394629 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } }
I20260928 05:11:25.395002 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: LEARNER
I20260928 05:11:25.395108 7199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:25.402107 7213 catalog_manager.cc:1259] Loaded cluster ID: 3a2752c27580474ca021d7bf60aa5459
I20260928 05:11:25.402220 7213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: loading cluster ID for follower catalog manager: success
I20260928 05:11:25.405609 7213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: acquiring CA information for follower catalog manager: success
I20260928 05:11:25.407859 7213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:11:25.553171 7021 heartbeater.cc:503] Master 127.0.97.62:43605 was elected leader, sending a full tablet report...
W20260928 05:11:25.581862 7127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:25.582310 7127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:25.582487 7127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:25.593003 7127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:25.593158 7127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:25.593241 7127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:25.593320 7127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:25.593384 7127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:25.603590 7127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:25.604843 7127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:25.606058 7127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:25.618265 7221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:25.618348 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:25.618319 7218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:25.619122 7127 server_base.cc:1034] running on GCE node
I20260928 05:11:25.619954 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:25.621366 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:25.622648 7127 hybrid_clock.cc:648] HybridClock initialized: now 1790572285622593 us; error 43 us; skew 500 ppm
I20260928 05:11:25.623000 7127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572285, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:25.626613 7127 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 05:11:25.628165 7127 webserver.cc:460] Webserver started at http://127.0.97.60:38131/ using document root <none> and password file <none>
I20260928 05:11:25.628836 7127 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:25.628973 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:25.629277 7127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:25.632288 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/instance:
uuid: "2295905206ad47ac84aae1ef0b8739db"
format_stamp: "Formatted at 2026-09-28 05:11:25 on dist-test-slave-6718"
server_key: "33801cf68f8e541d7207afc9d9dd3aaf"
server_key_iv: "b6f6ade82aa86a8157bcde39e3ee2c07"
server_key_version: "encryptionkey@0"
I20260928 05:11:25.633065 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal/instance:
uuid: "2295905206ad47ac84aae1ef0b8739db"
format_stamp: "Formatted at 2026-09-28 05:11:25 on dist-test-slave-6718"
server_key: "33801cf68f8e541d7207afc9d9dd3aaf"
server_key_iv: "b6f6ade82aa86a8157bcde39e3ee2c07"
server_key_version: "encryptionkey@0"
I20260928 05:11:25.638458 7127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 05:11:25.642364 7228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:25.643857 7127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:11:25.644047 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
uuid: "2295905206ad47ac84aae1ef0b8739db"
format_stamp: "Formatted at 2026-09-28 05:11:25 on dist-test-slave-6718"
server_key: "33801cf68f8e541d7207afc9d9dd3aaf"
server_key_iv: "b6f6ade82aa86a8157bcde39e3ee2c07"
server_key_version: "encryptionkey@0"
I20260928 05:11:25.644258 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:25.661979 7127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:25.666122 7127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:25.666390 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:25.685407 7127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45951
I20260928 05:11:25.685451 7279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45951 every 8 connection(s)
I20260928 05:11:25.686864 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:11:25.687263 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 7127
I20260928 05:11:25.690994 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:25.699520 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572285, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:25.713056 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.689777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.0.97.60:45951)
0928 05:11:25.690697 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.690712 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.691076 (+ 364us) server_negotiation.cc:407] Connection header received
0928 05:11:25.693390 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.693418 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.693911 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.694352 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.695840 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.697935 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.699302 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.700357 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.700535 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.705626 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.705633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.705643 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.705696 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.709020 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.709824 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.709835 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.709844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.710042 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.710456 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.710460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.710462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.710999 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.711290 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.711709 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.712094 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":380,"threads_started":1}
I20260928 05:11:25.717460 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.702099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:36220)
0928 05:11:25.702805 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.702846 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.702989 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.703367 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.703371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.703938 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.704251 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.704267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.705314 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.705319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.706581 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.706593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.707586 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.707595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.707756 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.708375 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.708393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.710772 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.713661 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.713673 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.713679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.714260 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.715986 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.715990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.715992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.716109 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.716629 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.716634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.716770 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.716946 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.717189 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":298,"threads_started":1}
I20260928 05:11:25.717873 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.702256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.0.97.62:43605)
0928 05:11:25.702454 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.702462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.702938 (+ 476us) server_negotiation.cc:407] Connection header received
0928 05:11:25.703115 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.703123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.703211 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.703334 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.704424 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.705161 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.706724 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.707401 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.707493 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.710937 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.710942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.710945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.710976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.713539 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.715685 (+ 2146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.715689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.715691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.715846 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.716216 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.716219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.716221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.716466 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.716576 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.717126 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.717525 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:11:25.719117 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572285, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:11:25.731740 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.720314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:51784)
0928 05:11:25.720502 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.720526 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.720638 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.721017 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.721020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.721319 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.721446 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.721453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.722691 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.722697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.723662 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.723673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.724281 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.724286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.724404 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.724929 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.724945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.726958 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.729745 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.729749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.729751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.730109 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.730441 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.730445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.730447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.730638 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.731063 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.731066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.731192 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.731303 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.731504 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:11:25.731828 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.720330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.97.61:34975)
0928 05:11:25.720776 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.720780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.720795 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:11:25.720839 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.720843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.720896 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.720980 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.721564 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.722569 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.723790 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.724155 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.724266 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.727135 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.727139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.727143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.727173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.729624 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.730216 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.730219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.730221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.730346 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.730727 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.730731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.730733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.730967 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.731081 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.731386 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.731533 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":218,"threads_started":1}
I20260928 05:11:25.732887 7280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572285, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 05:11:25.745632 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.733551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.0.1:50328)
0928 05:11:25.733769 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.733788 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.733902 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.734194 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.734198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.734588 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.734719 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.734727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.735663 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.735670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.736485 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.736495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.737164 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.737170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.737340 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.738248 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.738266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.740263 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.743164 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.743168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.743170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.743502 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.743917 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.743922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.743926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.744124 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.744721 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.744724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.744895 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.745059 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.745338 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:11:25.745949 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.733657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.0.97.60:45951)
0928 05:11:25.733797 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.733804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.733853 (+ 49us) server_negotiation.cc:407] Connection header received
0928 05:11:25.734007 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.734011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.734089 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.734182 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.734836 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.735512 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.736640 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.737016 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.737152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.740415 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.740420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.740429 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.740455 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.743024 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.743616 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.743622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.743626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.743789 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.744248 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.744252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.744255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.744562 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.744731 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.745152 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.745566 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":51}
I20260928 05:11:25.750034 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.744265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.0.97.61:34975)
0928 05:11:25.744446 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.744451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.744521 (+ 70us) server_negotiation.cc:407] Connection header received
0928 05:11:25.744692 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.744698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.744812 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.744907 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:25.745450 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.746494 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.748207 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.749149 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.749307 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.749520 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.749673 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.749807 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:11:25.750097 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.744005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:51798)
0928 05:11:25.744451 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.744466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.744565 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.744968 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.744973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.745003 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:25.745215 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.745229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.746648 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.746652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.748053 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.748066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.749317 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.749326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.749485 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.749539 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.749652 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.749787 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":217,"threads_started":1}
I20260928 05:11:25.751245 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Bootstrap starting.
I20260928 05:11:25.751232 7174 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 05:11:25.751780 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540972e76935476f91b35f900d41dae7. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } }
I20260928 05:11:25.752031 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: LEARNER
I20260928 05:11:25.755369 7280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:25.756419 7280 log.cc:824] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:25.759101 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: No bootstrap required, opened a new log
I20260928 05:11:25.765892 7280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } }
I20260928 05:11:25.766215 7280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:25.766324 7280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2295905206ad47ac84aae1ef0b8739db, State: Initialized, Role: FOLLOWER
I20260928 05:11:25.767014 7280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } }
I20260928 05:11:25.767875 7289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } } }
I20260928 05:11:25.768594 7289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:25.768929 7280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:11:25.777590 7300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:11:25.777741 7300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:11:25.781458 7127 master_runner.cc:428] Detected that this master 2295905206ad47ac84aae1ef0b8739db is joining an existing cluster
I20260928 05:11:25.781548 7127 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:45951
I20260928 05:11:25.791369 7289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:25.791623 7289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } }
I20260928 05:11:25.793069 7289 leader_election.cc:288] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605), 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61:34975)
I20260928 05:11:25.794164 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2295905206ad47ac84aae1ef0b8739db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540972e76935476f91b35f900d41dae7" is_pre_election: true
I20260928 05:11:25.794747 7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2295905206ad47ac84aae1ef0b8739db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b19a82457942dc91b0ee156ee92f21" is_pre_election: true
I20260928 05:11:25.796135 7232 leader_election.cc:302] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2295905206ad47ac84aae1ef0b8739db; no voters: 50b19a82457942dc91b0ee156ee92f21, 540972e76935476f91b35f900d41dae7
I20260928 05:11:25.797024 7289 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 05:11:25.802978 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.785193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.0.97.62:43605)
0928 05:11:25.785597 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.785601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.786663 (+ 1062us) server_negotiation.cc:407] Connection header received
0928 05:11:25.786969 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.786976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.787074 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.787156 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.788960 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.790405 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.792567 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.793185 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.793296 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.795996 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.796002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.796007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.796068 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.799703 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.800609 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.800616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.800622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.800770 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.801254 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.801261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.801266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.801580 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.801713 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.802334 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.802599 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260928 05:11:25.804589 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.785690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:36236)
0928 05:11:25.786520 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.786567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.786797 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.787419 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.787427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.788293 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.788756 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.788771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.790648 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.790657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.792406 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.792418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.793569 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.793574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.793755 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.794474 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.794502 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.795749 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.799875 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.799888 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.799892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.800453 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.800905 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.800912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.800916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.801100 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.801788 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.801792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.802003 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.803423 (+ 1420us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.803728 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":438,"threads_started":1}
I20260928 05:11:25.812062 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.785363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.97.61:34975)
0928 05:11:25.785696 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.785708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.793964 (+ 8256us) server_negotiation.cc:407] Connection header received
0928 05:11:25.794309 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.794315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.794390 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.794463 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.795991 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.797463 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.799477 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.800106 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.800235 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.802748 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.802754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.802760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.802799 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.808841 (+ 6042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.809721 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.809728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.809732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.809879 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.810349 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.810355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.810359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.810727 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.810861 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.811448 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.811700 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260928 05:11:25.813282 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.785847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:51812)
0928 05:11:25.793858 (+ 8011us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.793876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.794157 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.794696 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.794703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.795444 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.795824 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.795837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.797666 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.797671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.799308 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.799322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.800478 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.800485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.800715 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.801349 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.801371 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.802506 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.809012 (+ 6506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.809019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.809024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.809572 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.810023 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.810030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.810036 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.810217 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.810926 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.810931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.811151 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.812481 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.812806 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7879,"thread_start_us":13024,"threads_started":1}
I20260928 05:11:25.814847 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.786848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.0.97.60:45951)
0928 05:11:25.787045 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.787051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.798867 (+ 11816us) server_negotiation.cc:407] Connection header received
0928 05:11:25.799243 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.799249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.799313 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.799374 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.800836 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.801915 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.803875 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.804962 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.805098 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.807026 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.807031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.807036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.807071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.810776 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.812425 (+ 1649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.812430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.812433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.812576 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.812964 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.812969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.812972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.813305 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.813462 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.813925 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.814497 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:11:25.815950 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.787143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.0.1:50334)
0928 05:11:25.798780 (+ 11637us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.798798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.799114 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.799555 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.799559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.800293 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.800668 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.800681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.802115 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.802123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.803732 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.803744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.805137 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.805143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.805302 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.805801 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.805815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.806836 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.810894 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.810899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.810904 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.811383 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.812707 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.812714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.812719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.812850 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.813518 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.813522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.813679 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.815501 (+ 1822us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.815673 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11480,"thread_start_us":6842,"threads_started":1}
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 05:11:25.824008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6626
I20260928 05:11:25.828610 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.810130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.0.1:50350)
0928 05:11:25.810430 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.810448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.810576 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.815397 (+ 4821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.815404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.815864 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.816042 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.816053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.817689 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.817694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.818630 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.818645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.819502 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.819509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.819681 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.820215 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.820232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.822087 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.825722 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.825728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.825734 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.826221 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.826666 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.826670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.826673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.826835 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.827636 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.827641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.827807 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.827943 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.828211 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260928 05:11:25.828626 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.810781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.0.97.60:45951)
0928 05:11:25.815064 (+ 4283us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.815069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.815084 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:11:25.815161 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.815166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.815253 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.815358 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.816165 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.817523 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.818846 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.819350 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.819491 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.822319 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.822325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.822330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.822365 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.825556 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.826367 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.826371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.826375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.826501 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.826962 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.826965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.826967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.827515 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.827656 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.828020 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.828221 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4176,"thread_start_us":11951,"threads_started":1}
I20260928 05:11:25.829996 6565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:45951: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2295905206ad47ac84aae1ef0b8739db"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 45951
}
attrs {
promote: true
}
}
dest_uuid: "540972e76935476f91b35f900d41dae7"
cas_config_opid_index: 5
I20260928 05:11:25.831055 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } }
I20260928 05:11:25.832860 7286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2295905206ad47ac84aae1ef0b8739db (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.836634 7286 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:45951
Sep 28 05:11:25 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 05:11:25.838321 7255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:25.841909 7255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Refusing update from remote peer 540972e76935476f91b35f900d41dae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 05:11:25.842396 7127 master.cc:537] Master@127.0.97.60:45951 shutting down...
I20260928 05:11:25.843850 7174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 LEARNER]: Refusing update from remote peer 540972e76935476f91b35f900d41dae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 05:11:25.850090 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.825376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.0.97.1:45689)
0928 05:11:25.826066 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.826076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.826107 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:11:25.826172 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.826180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.826466 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.826659 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.829008 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.831147 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.832832 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.834081 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.835054 (+ 973us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.839744 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.839764 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.839787 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.839853 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.844558 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.845647 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.845660 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.845666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.845879 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.846397 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.846404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.846410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.846988 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.848914 (+ 1926us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.849121 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.849450 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":336,"threads_started":1}
I20260928 05:11:25.852140 6749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:25.864749 7311 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 05:11:25.872898 7286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:25.873128 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.873528 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:25.879148 7174 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2295905206ad47ac84aae1ef0b8739db (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.883708 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: true } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.883992 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: LEARNER
I20260928 05:11:25.885003 7127 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:25.885378 7127 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:25.885540 7127 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: stopping tablet replica
I20260928 05:11:25.885679 7286 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: attempting to promote NON_VOTER 50b19a82457942dc91b0ee156ee92f21 to VOTER
I20260928 05:11:25.886547 7286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } }
I20260928 05:11:25.891759 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 LEARNER]: Refusing update from remote peer 540972e76935476f91b35f900d41dae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 05:11:25.891916 7127 master.cc:552] Master@127.0.97.60:45951 shutdown complete.
I20260928 05:11:25.891980 7127 master_runner.cc:305] Clearing existing system tablet
I20260928 05:11:25.892342 7127 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:25.892464 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:25.893079 7311 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 05:11:25.898360 7127 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:11:25.898690 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.897819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.0.1:50362)
0928 05:11:25.898006 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.898143 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 05:11:25.899394 6549 consensus_peers.cc:487] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 -> Peer 2295905206ad47ac84aae1ef0b8739db (127.0.97.60:45951): Couldn't send request to peer 2295905206ad47ac84aae1ef0b8739db. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45951: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 05:11:25.898972 7312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.906883 7320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:25.907176 7127 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:25.907493 7127 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.002s
I20260928 05:11:25.907588 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
uuid: "2295905206ad47ac84aae1ef0b8739db"
format_stamp: "Formatted at 2026-09-28 05:11:25 on dist-test-slave-6718"
server_key: "33801cf68f8e541d7207afc9d9dd3aaf"
server_key_iv: "b6f6ade82aa86a8157bcde39e3ee2c07"
server_key_version: "encryptionkey@0"
I20260928 05:11:25.912820 7127 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:25.919520 7127 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 05:11:25.919761 7127 log.cc:1196] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 05:11:25.920284 7127 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Deleting consensus metadata
I20260928 05:11:25.920560 7127 master_runner.cc:315] Copying system tablet from 127.0.97.62:43605
I20260928 05:11:25.923856 7174 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.929214 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.929477 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:25.930485 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.930796 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:25.934708 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:25.934973 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:25.949036 7127 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43605
I20260928 05:11:25.965904 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.949848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.97.62:43605)
0928 05:11:25.950078 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:25.950085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:25.951112 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 05:11:25.951565 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:25.951571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:25.951652 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:25.951755 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:25.952933 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.954102 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.956247 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.956860 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.956982 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.959538 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:25.959545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:25.959550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:25.959588 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:25.962974 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.963786 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.963792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.963796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.963931 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:25.964356 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:25.964361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:25.964365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:25.964685 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:25.964829 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:25.965347 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 05:11:25.965552 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 05:11:25.966316 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:25.949832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:36250)
0928 05:11:25.950989 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:25.951017 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:25.951296 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:25.951809 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:25.951816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:25.952475 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:25.952780 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.952793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.954238 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.954245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.956077 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:25.956092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:25.957016 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:25.957023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:25.957244 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:25.958105 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:25.958139 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:25.959349 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:25.963121 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.963135 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.963140 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.963650 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.964045 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:25.964050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:25.964055 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:25.964239 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:25.964842 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:25.964848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:25.965043 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 05:11:25.965209 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 05:11:25.965542 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":824,"threads_started":1}
I20260928 05:11:25.967556 6585 tablet_copy_service.cc:140] P 540972e76935476f91b35f900d41dae7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2295905206ad47ac84aae1ef0b8739db ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36250)
I20260928 05:11:25.967742 6585 tablet_copy_service.cc:161] P 540972e76935476f91b35f900d41dae7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2295905206ad47ac84aae1ef0b8739db at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36250: session id = 2295905206ad47ac84aae1ef0b8739db-00000000000000000000000000000000
I20260928 05:11:25.971836 6585 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 05:11:25.973624 7127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 05:11:25.980147 7127 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: tablet copy: Starting download of 0 data blocks...
I20260928 05:11:25.980435 7127 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: tablet copy: Starting download of 1 WAL segments...
I20260928 05:11:25.983352 7127 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 05:11:25.987635 6585 tablet_copy_service.cc:342] P 540972e76935476f91b35f900d41dae7: Request end of tablet copy session 2295905206ad47ac84aae1ef0b8739db-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36250
I20260928 05:11:25.987800 6585 tablet_copy_service.cc:433] P 540972e76935476f91b35f900d41dae7: ending tablet copy session 2295905206ad47ac84aae1ef0b8739db-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2295905206ad47ac84aae1ef0b8739db
I20260928 05:11:26.000589 7127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:26.003530 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:26.005103 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:26.008066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6626
W20260928 05:11:26.030516 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:26.030720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45689
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43605
--webserver_port=44321
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:11:26.031136 7127 server_base.cc:1034] running on GCE node
I20260928 05:11:26.031464 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:26.032143 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:26.033408 7127 hybrid_clock.cc:648] HybridClock initialized: now 1790572286033371 us; error 32 us; skew 500 ppm
I20260928 05:11:26.033726 7127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:26 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:26.036036 7127 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 05:11:26.037436 7127 webserver.cc:460] Webserver started at http://127.0.97.60:42859/ using document root <none> and password file <none>
I20260928 05:11:26.037750 7127 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:26.037842 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:26.040911 7127 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:26.042961 7338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:26.043385 7127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 05:11:26.043501 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
uuid: "2295905206ad47ac84aae1ef0b8739db"
format_stamp: "Formatted at 2026-09-28 05:11:25 on dist-test-slave-6718"
server_key: "33801cf68f8e541d7207afc9d9dd3aaf"
server_key_iv: "b6f6ade82aa86a8157bcde39e3ee2c07"
server_key_version: "encryptionkey@0"
I20260928 05:11:26.043679 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:26.074394 7127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:26.078577 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:26.098160 7127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45951
I20260928 05:11:26.098201 7389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45951 every 8 connection(s)
I20260928 05:11:26.099226 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:11:26.104249 7390 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:11:26.108286 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Bootstrap starting.
I20260928 05:11:26.133131 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:11:26.133661 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Bootstrap complete.
I20260928 05:11:26.134698 7390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } }
I20260928 05:11:26.134980 7390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2295905206ad47ac84aae1ef0b8739db, State: Initialized, Role: LEARNER
I20260928 05:11:26.135224 7390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } }
I20260928 05:11:26.136317 7390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:26.136132 7393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:26.136598 7393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: This master's current role is: LEARNER
I20260928 05:11:26.147539 7404 catalog_manager.cc:1259] Loaded cluster ID: 3a2752c27580474ca021d7bf60aa5459
I20260928 05:11:26.147641 7404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: loading cluster ID for follower catalog manager: success
I20260928 05:11:26.150884 7404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: acquiring CA information for follower catalog manager: success
I20260928 05:11:26.153076 7404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:11:26.390455 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:26.390844 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:26.391016 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:26.400372 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:26.400535 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:26.400709 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:26.411396 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44321
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:26.412700 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:26.413982 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:26.427448 7410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:26.427384 7409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:26.427484 7412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:26.428632 7333 server_base.cc:1034] running on GCE node
I20260928 05:11:26.429260 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:26.430163 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:26.431391 7333 hybrid_clock.cc:648] HybridClock initialized: now 1790572286431326 us; error 60 us; skew 500 ppm
I20260928 05:11:26.431730 7333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:26 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:26.435406 7333 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:11:26.436975 7333 webserver.cc:460] Webserver started at http://127.0.97.1:44321/ using document root <none> and password file <none>
I20260928 05:11:26.437726 7333 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:26.437893 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:26.443734 7333 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 05:11:26.446627 7419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:26.447968 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:11:26.448148 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "5e0c6f7dbe174d28976b26e4ba2be73a"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "5b5985f037183486ccfc25b7b15ad3c0"
server_key_iv: "5096e8ce4186bac6fd5948b06f748467"
server_key_version: "encryptionkey@0"
I20260928 05:11:26.449177 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:26.470703 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:26.474838 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:26.475117 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:26.476106 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:26.477531 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:26.477650 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:26.477785 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:26.477870 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:26.491189 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.484082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.0.1:47356)
0928 05:11:26.484592 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:26.484619 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:26.484765 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:26.485452 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:26.485458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:26.485490 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:26.485718 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.485734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.487508 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.487512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.489016 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.489037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.490596 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.490604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.490751 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.490777 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:11:26.490854 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:11:26.490950 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260928 05:11:26.491541 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.484249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.0.97.60:45951)
0928 05:11:26.484982 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:26.484991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:26.485019 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:11:26.485116 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:26.485124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:26.485239 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:26.485400 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:26.485931 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.487320 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.489183 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.490345 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.490508 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.490865 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:26.490933 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 05:11:26.491114 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":374,"threads_started":1}
I20260928 05:11:26.492677 7365 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 05:11:26.493330 7393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540972e76935476f91b35f900d41dae7. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:26.493657 7393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: This master's current role is: LEARNER
I20260928 05:11:26.515640 7333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45689
I20260928 05:11:26.515720 7534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45689 every 8 connection(s)
I20260928 05:11:26.517050 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:26.523113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 7333
I20260928 05:11:26.523475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6760
Sep 28 05:11:26 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 05:11:26 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 05:11:26 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:11:26 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 05:11:26.557230 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.535106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56991 (local address 127.0.97.60:45951)
0928 05:11:26.535270 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:26.535277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:26.539805 (+ 4528us) server_negotiation.cc:407] Connection header received
0928 05:11:26.540028 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:26.540034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:26.540101 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:26.540161 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:26.541670 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.543243 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.545180 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.545882 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.545982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.550455 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:26.550464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:26.550470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:26.550510 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:26.554208 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.555052 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.555057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.555061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.555211 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.555683 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.555689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.555693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.556016 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:26.556152 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:26.556615 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 05:11:26.556832 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:11:26.558439 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.521071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.97.1:41713)
0928 05:11:26.531362 (+ 10291us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:26.531382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:26.531470 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:26.536087 (+ 4617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:26.536125 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:26.537469 (+ 1344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:26.538601 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.538638 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.540438 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.540450 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.542778 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.542792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.544002 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.544009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.544361 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.545181 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:26.545212 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:26.548529 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:26.553332 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:26.553343 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:26.553361 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:26.554134 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:26.554816 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:26.554821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:26.554826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:26.555059 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:26.555762 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:26.555772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:26.556178 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 05:11:26.556501 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 05:11:26.557013 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10160,"mutex_wait_us":86,"thread_start_us":5674,"threads_started":1}
I20260928 05:11:26.559037 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.520044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35745 (local address 127.0.97.62:43605)
0928 05:11:26.539446 (+ 19402us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:26.539452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:26.539471 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:11:26.539527 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:26.539533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:26.539606 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:26.539725 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:26.541344 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.542701 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.544741 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.545528 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.545660 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.549844 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:26.549851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:26.549856 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:26.549902 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:26.555277 (+ 5375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.556065 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.556070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.556077 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.556219 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.556650 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.556654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.556657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.556987 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:26.557103 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:26.558416 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0928 05:11:26.558656 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19256,"thread_start_us":19125,"threads_started":1}
I20260928 05:11:26.559058 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.520041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41713 (local address 127.0.97.61:34975)
0928 05:11:26.535245 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:26.535251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:26.535280 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:11:26.535344 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:26.535349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:26.535461 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:26.538844 (+ 3383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:26.539107 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.540266 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.542940 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.543585 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.543702 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.548813 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:26.548820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:26.548825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:26.548887 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:26.553194 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.554331 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.554336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.554340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.554512 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.555192 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.555197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.555200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.555553 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:26.555683 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:26.558277 (+ 2594us) server_negotiation.cc:299] Negotiation successful
0928 05:11:26.558500 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15085,"thread_start_us":15554,"threads_started":1}
I20260928 05:11:26.559410 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.520579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:35745)
0928 05:11:26.527156 (+ 6577us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:26.527221 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:26.530446 (+ 3225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:26.539748 (+ 9302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:26.539752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:26.540193 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:26.540496 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.540515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.542875 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.542880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.544540 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.544554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.545878 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.545884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.546048 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.546730 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:26.546751 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:26.549615 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:26.555451 (+ 5836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:26.555456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:26.555459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:26.555940 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:26.556356 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:26.556361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:26.556364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:26.556522 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:26.557225 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:26.557232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:26.557399 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 05:11:26.557543 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:11:26.557816 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6263,"mutex_wait_us":118,"thread_start_us":10382,"threads_started":1}
I20260928 05:11:26.559868 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.535107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.97.1:56991)
0928 05:11:26.539678 (+ 4571us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:26.539700 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:26.539894 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:26.540352 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:26.540356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:26.541018 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:26.541490 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.541504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.543428 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.543433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.545013 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:26.545026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.546277 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.546283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:26.546508 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.547167 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:26.547194 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:26.550231 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:26.554346 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:26.554351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:26.554354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:26.554920 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:26.555355 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:26.555360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:26.555363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:26.555548 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:26.556228 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:26.556233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:26.556356 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 05:11:26.557756 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0928 05:11:26.558017 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4446,"mutex_wait_us":94,"thread_start_us":4259,"threads_started":1}
I20260928 05:11:26.560561 7536 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43605
I20260928 05:11:26.560716 7537 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34975
I20260928 05:11:26.560900 7537 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:26.560905 7536 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:26.561273 7535 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45951
I20260928 05:11:26.561401 7535 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:26.562410 7535 heartbeater.cc:511] Master 127.0.97.60:45951 requested a full tablet report, sending...
I20260928 05:11:26.562439 7537 heartbeater.cc:511] Master 127.0.97.61:34975 requested a full tablet report, sending...
I20260928 05:11:26.562495 7536 heartbeater.cc:511] Master 127.0.97.62:43605 requested a full tablet report, sending...
I20260928 05:11:26.562655 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:26.524595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.0.97.2:38161)
0928 05:11:26.525710 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:26.525723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:26.535214 (+ 9491us) server_negotiation.cc:407] Connection header received
0928 05:11:26.535560 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:26.535574 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:26.535927 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:26.536097 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:26.540122 (+ 4025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.541441 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.543711 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:26.544583 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:26.544705 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:26.551069 (+ 6364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:26.551083 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:26.551101 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:26.551164 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:26.557460 (+ 6296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.558377 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.558385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.558389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.558554 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:26.559066 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:26.559072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:26.559075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:26.559498 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:26.559968 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:26.561522 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0928 05:11:26.562222 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":777,"threads_started":1}
I20260928 05:11:26.563932 6883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:26.564939 7355 ts_manager.cc:194] Registered new tserver with Master: 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689)
I20260928 05:11:26.564939 7164 ts_manager.cc:194] Registered new tserver with Master: 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689)
I20260928 05:11:26.565871 6565 ts_manager.cc:194] Re-registered known tserver with Master: 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689)
I20260928 05:11:26.566715 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35745
I20260928 05:11:26.650521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6760
I20260928 05:11:26.667742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38161
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43605
--webserver_port=43385
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:26.998611 7547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:26.998889 7547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:26.999046 7547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:27.006592 7547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:27.006727 7547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:27.006872 7547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:27.016613 7547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43385
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:27.017899 7547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:27.019312 7547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:27.032488 7552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:27.032545 7553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:27.032673 7555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:27.033464 7547 server_base.cc:1034] running on GCE node
I20260928 05:11:27.034155 7547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:27.035202 7547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:27.036443 7547 hybrid_clock.cc:648] HybridClock initialized: now 1790572287036401 us; error 39 us; skew 500 ppm
I20260928 05:11:27.036779 7547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:27.040000 7547 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:11:27.041472 7547 webserver.cc:460] Webserver started at http://127.0.97.2:43385/ using document root <none> and password file <none>
I20260928 05:11:27.042140 7547 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:27.042327 7547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:27.048305 7547 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:11:27.051064 7562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:27.052495 7547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:27.052719 7547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "d441a081acef4a40800dd49ad09f5f29"
format_stamp: "Formatted at 2026-09-28 05:11:23 on dist-test-slave-6718"
server_key: "d1825cca9006d7c50c53b8940c3bdd18"
server_key_iv: "3ca73089c4a8a3c59143421f55be4791"
server_key_version: "encryptionkey@0"
I20260928 05:11:27.053839 7547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:27.059216 7424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: true } } }
I20260928 05:11:27.059648 7424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:27.061300 7424 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: attempting to promote NON_VOTER 2295905206ad47ac84aae1ef0b8739db to VOTER
I20260928 05:11:27.061908 7424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } }
I20260928 05:11:27.063777 7174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Refusing update from remote peer 540972e76935476f91b35f900d41dae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 05:11:27.063782 7365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 LEARNER]: Refusing update from remote peer 540972e76935476f91b35f900d41dae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 05:11:27.064340 7424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 05:11:27.064728 7564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 05:11:27.067374 7424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2295905206ad47ac84aae1ef0b8739db (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.068082 7174 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2295905206ad47ac84aae1ef0b8739db (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.072163 7563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.072404 7563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:27.072794 7563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.072444 7365 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2295905206ad47ac84aae1ef0b8739db (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.072963 7563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:27.073264 7563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.073432 7563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:27.074640 7573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.074887 7573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:27.078123 7574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "540972e76935476f91b35f900d41dae7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:27.078490 7574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:27.080858 7547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:27.085096 7547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:27.085364 7547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:27.086331 7547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:27.087731 7547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:27.087838 7547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:27.087960 7547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:27.088052 7547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:27.118932 7547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38161
I20260928 05:11:27.118978 7682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38161 every 8 connection(s)
I20260928 05:11:27.120293 7547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:27.127164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 7547
I20260928 05:11:27.127465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6893
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 05:11:27.172467 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.128527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.0.97.3:37397)
0928 05:11:27.146104 (+ 17577us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.146117 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.146152 (+ 35us) server_negotiation.cc:407] Connection header received
0928 05:11:27.146230 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.146239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.146519 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.149282 (+ 2763us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.149582 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.151675 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.155106 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.156260 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.156412 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.164305 (+ 7893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.164320 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.164337 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.164375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.168322 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.169169 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.169181 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.169185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.169348 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.169815 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.169820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.169823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.170206 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.170476 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.171742 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.172026 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17357,"thread_start_us":18157,"threads_started":1}
I20260928 05:11:27.174763 7016 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:27.185063 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.123315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52967 (local address 127.0.97.61:34975)
0928 05:11:27.152730 (+ 29415us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.152739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.162860 (+ 10121us) server_negotiation.cc:407] Connection header received
0928 05:11:27.162924 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.162929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.163001 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.163101 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.166335 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.167972 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.170563 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.171183 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.171319 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.177553 (+ 6234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.177561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.177566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.177611 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.181155 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.182061 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.182069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.182075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.182239 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.182713 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.182718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.182721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.183086 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.183217 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.184338 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.184575 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29240,"thread_start_us":29438,"threads_started":1}
I20260928 05:11:27.186555 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.123324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35901 (local address 127.0.97.60:45951)
0928 05:11:27.158734 (+ 35410us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.158742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.158766 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:27.162010 (+ 3244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.162016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.162114 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.162201 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.164905 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.167621 (+ 2716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.170535 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.171217 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.171324 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.176976 (+ 5652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.176986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.176991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.177037 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.180752 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.181841 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.181847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.181852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.182003 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.182625 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.182632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.182638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.183118 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.183277 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.185706 (+ 2429us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.186033 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22897,"thread_start_us":31740,"threads_started":1}
I20260928 05:11:27.187065 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.123913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.97.2:52967)
0928 05:11:27.162607 (+ 38694us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.162626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.162739 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.165220 (+ 2481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.165225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.165778 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:27.166063 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.166076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.168171 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.168178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.170330 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.170344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.171655 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.171663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.172090 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.173008 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:27.173027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:27.177331 (+ 4304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:27.181316 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.181321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.181325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.181918 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.182386 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.182393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.182399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.182590 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.183327 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:27.183335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:27.183858 (+ 523us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.185285 (+ 1427us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.185572 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38516,"mutex_wait_us":340,"spinlock_wait_cycles":43648,"thread_start_us":38221,"threads_started":1}
I20260928 05:11:27.187664 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.123913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.97.2:35901)
0928 05:11:27.147313 (+ 23400us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.147380 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.161644 (+ 14264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.162869 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.162883 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.163759 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:27.164683 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.164712 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.167837 (+ 3125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.167843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.170331 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.170344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.171655 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.171663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.172086 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.172989 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:27.173019 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:27.176210 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:27.180951 (+ 4741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.180962 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.180983 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.181666 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.182152 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.182156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.182160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.182431 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.183363 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:27.183371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:27.183876 (+ 505us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.184222 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.184956 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23038,"mutex_wait_us":244,"spinlock_wait_cycles":11520,"thread_start_us":22816,"threads_started":1}
I20260928 05:11:27.188903 7683 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45951
I20260928 05:11:27.188946 7685 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34975
I20260928 05:11:27.189244 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.148517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58005 (local address 127.0.97.62:43605)
0928 05:11:27.159457 (+ 10940us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.159463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.165346 (+ 5883us) server_negotiation.cc:407] Connection header received
0928 05:11:27.165631 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.165637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.165726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.165809 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.167617 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.168884 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.170885 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.171547 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.171641 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.182736 (+ 11095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.182744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.182748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.182797 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.186304 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.187054 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.187062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.187065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.187225 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.187691 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.187698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.187701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.188045 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.188178 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.188782 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.188992 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10798,"thread_start_us":10945,"threads_started":1}
I20260928 05:11:27.189252 7685 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:27.189250 7683 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:27.190239 7683 heartbeater.cc:511] Master 127.0.97.60:45951 requested a full tablet report, sending...
I20260928 05:11:27.190261 7685 heartbeater.cc:511] Master 127.0.97.61:34975 requested a full tablet report, sending...
I20260928 05:11:27.191088 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.148461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:58005)
0928 05:11:27.165232 (+ 16771us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.165250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.165465 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.166067 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.166073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.166701 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:27.167373 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.167386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.169070 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.169075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.170723 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.170736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.171942 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.171950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.172169 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.173196 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:27.173229 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:27.182153 (+ 8924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:27.186420 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.186425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.186428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.186928 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.187349 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.187354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.187357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.187564 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.188372 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:27.188377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:27.188480 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.188642 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.190719 (+ 2077us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16640,"mutex_wait_us":59,"spinlock_wait_cycles":32384,"thread_start_us":16494,"threads_started":1}
I20260928 05:11:27.192154 7355 ts_manager.cc:194] Registered new tserver with Master: d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161)
I20260928 05:11:27.192557 7164 ts_manager.cc:194] Registered new tserver with Master: d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161)
I20260928 05:11:27.192579 7684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43605
I20260928 05:11:27.192703 7684 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:27.193143 7684 heartbeater.cc:511] Master 127.0.97.62:43605 requested a full tablet report, sending...
I20260928 05:11:27.194027 6565 ts_manager.cc:194] Re-registered known tserver with Master: d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161)
I20260928 05:11:27.194990 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58005
I20260928 05:11:27.272511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6893
I20260928 05:11:27.288380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37397
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43605
--webserver_port=45127
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:11:27.569314 7536 heartbeater.cc:503] Master 127.0.97.62:43605 was elected leader, sending a full tablet report...
W20260928 05:11:27.608917 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:27.609215 7696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:27.609373 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:27.616676 7696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:27.616827 7696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:27.617004 7696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:27.626547 7696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45127
--tserver_master_addrs=127.0.97.62:43605,127.0.97.61:34975,127.0.97.60:45951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:27.627991 7696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:27.629243 7696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:27.641764 7701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:27.641809 7702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:27.641835 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:27.642896 7696 server_base.cc:1034] running on GCE node
I20260928 05:11:27.643573 7696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:27.644562 7696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:27.645788 7696 hybrid_clock.cc:648] HybridClock initialized: now 1790572287645739 us; error 42 us; skew 500 ppm
I20260928 05:11:27.646093 7696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:27.649281 7696 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:11:27.650658 7696 webserver.cc:460] Webserver started at http://127.0.97.3:45127/ using document root <none> and password file <none>
I20260928 05:11:27.651271 7696 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:27.651402 7696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:27.657089 7696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 05:11:27.660013 7711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:27.661427 7696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 05:11:27.661605 7696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "1b03c854497442c99bc3a3dfcfaa3534"
format_stamp: "Formatted at 2026-09-28 05:11:24 on dist-test-slave-6718"
server_key: "b1b2f880c1956d91a587f340f2ab5810"
server_key_iv: "9b222fd02cfc5cfc00d287af888526c9"
server_key_version: "encryptionkey@0"
I20260928 05:11:27.662858 7696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:27.701694 7696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:27.705822 7696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:27.706099 7696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:27.707134 7696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:27.708732 7696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:27.708848 7696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:27.708966 7696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:27.709060 7696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:27.741554 7696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37397
I20260928 05:11:27.741638 7823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37397 every 8 connection(s)
I20260928 05:11:27.743028 7696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:27.747691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 7696
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:27.789984 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.746050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59479 (local address 127.0.97.60:45951)
0928 05:11:27.754937 (+ 8887us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.754945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.759457 (+ 4512us) server_negotiation.cc:407] Connection header received
0928 05:11:27.761397 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.761404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.761510 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.761625 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.764400 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.765929 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.769258 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.769922 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.770036 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.775693 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.775702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.775710 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.775763 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.779919 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.780885 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.780891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.780894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.782254 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.786940 (+ 4686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.786945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.786948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.787323 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.787463 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.789382 (+ 1919us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.789592 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8741,"thread_start_us":8639,"threads_started":1}
I20260928 05:11:27.791325 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.746623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.97.3:59479)
0928 05:11:27.759285 (+ 12662us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.759330 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.761122 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.762072 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.762084 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.763086 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:27.764125 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.764158 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.766266 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.766274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.769078 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.769094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.770352 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.770358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.770792 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.771593 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:27.771629 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:27.775355 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:27.780112 (+ 4757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.780121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.780139 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.780740 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.786060 (+ 5320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.786065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.786069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.786356 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.787579 (+ 1223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:27.787587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:27.788059 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.790543 (+ 2484us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.790829 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12368,"thread_start_us":14806,"threads_started":1}
I20260928 05:11:27.792439 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.762261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35059 (local address 127.0.97.62:43605)
0928 05:11:27.770240 (+ 7979us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.770246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.770265 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:11:27.770323 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.770328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.770412 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.771301 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.771833 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.773252 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.775757 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.776606 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.777339 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.782079 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.782087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.782095 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.782140 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.786975 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.787743 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.787747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.787750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.787918 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.788346 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.788352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.788357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.788787 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.791473 (+ 2686us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.791625 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.791930 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7858,"thread_start_us":7759,"threads_started":1}
I20260928 05:11:27.791332 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.762193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:35059)
0928 05:11:27.769157 (+ 6964us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.769186 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.769340 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.770608 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.770612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.771237 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:27.771636 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.771650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.773451 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.773459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.775521 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.775540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.776839 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.776848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.777047 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.777594 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:27.777616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:27.781866 (+ 4250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:27.787108 (+ 5242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.787113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.787116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.787602 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.788055 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.788058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.788062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.788205 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.788962 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:27.788970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:27.789141 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.789305 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.789823 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6775,"thread_start_us":7231,"threads_started":1}
Sep 28 05:11:27 dist-test-slave-6718 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 05:11:27.795787 7825 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43605
I20260928 05:11:27.796167 7825 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:27.797148 7825 heartbeater.cc:511] Master 127.0.97.62:43605 requested a full tablet report, sending...
I20260928 05:11:27.798022 7824 heartbeater.cc:347] Connected to a master server at 127.0.97.60:45951
I20260928 05:11:27.798148 7824 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:27.798578 7824 heartbeater.cc:511] Master 127.0.97.60:45951 requested a full tablet report, sending...
I20260928 05:11:27.799991 7355 ts_manager.cc:194] Registered new tserver with Master: 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397)
I20260928 05:11:27.800770 6565 ts_manager.cc:194] Re-registered known tserver with Master: 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397)
I20260928 05:11:27.801949 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35059
I20260928 05:11:27.811316 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.747209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.97.3:60365)
0928 05:11:27.761866 (+ 14657us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.761883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.768150 (+ 6267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.768820 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.768830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.769482 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:27.769733 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.769745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.771414 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.771423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.778755 (+ 7332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.778779 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.781296 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.781305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.781577 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.790450 (+ 8873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:27.790481 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:27.802710 (+ 12229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:27.807162 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.807169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.807175 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.807702 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.808687 (+ 985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:27.808694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:27.808698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:27.808845 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:27.810144 (+ 1299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:27.810149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:27.810301 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.810449 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.810800 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14524,"thread_start_us":11708,"threads_started":1}
I20260928 05:11:27.812602 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.746017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60365 (local address 127.0.97.61:34975)
0928 05:11:27.758731 (+ 12714us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.758741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.761954 (+ 3213us) server_negotiation.cc:407] Connection header received
0928 05:11:27.768429 (+ 6475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.768435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.768515 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.768591 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.770204 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.771247 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.778985 (+ 7738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.779872 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.779996 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.802878 (+ 22882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.802885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.802889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.802929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.806953 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.807867 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.807877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.807881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.808523 (+ 642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.808976 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.808982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.808988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.809337 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.809467 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.811985 (+ 2518us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.812198 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12567,"thread_start_us":12499,"threads_started":1}
I20260928 05:11:27.813726 7826 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34975
I20260928 05:11:27.813863 7826 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:27.814276 7826 heartbeater.cc:511] Master 127.0.97.61:34975 requested a full tablet report, sending...
I20260928 05:11:27.815441 7164 ts_manager.cc:194] Registered new tserver with Master: 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397)
I20260928 05:11:27.815660 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.771768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.97.62:43605)
0928 05:11:27.792992 (+ 21224us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.792998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.798294 (+ 5296us) server_negotiation.cc:407] Connection header received
0928 05:11:27.798547 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.798555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.798631 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.798697 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.801434 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.802422 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.804372 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.804946 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.805039 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.809151 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.809157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.809162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.809221 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.812584 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.813373 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.813379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.813385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.813544 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.814026 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.814032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.814035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.814400 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.814661 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.815062 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.815277 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21096,"thread_start_us":29444,"threads_started":1}
I20260928 05:11:27.819671 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.798171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.0.97.60:45951)
0928 05:11:27.801617 (+ 3446us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.801622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.801638 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:11:27.801684 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.801691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.801794 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.801888 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.805102 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.806121 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.808159 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.808681 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.808812 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.812287 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.812292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.812297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.812336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.816190 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.817038 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.817043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.817047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.817181 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.817614 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.817619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.817623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.818454 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.818742 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.819117 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.819349 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3329}
I20260928 05:11:27.821534 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.781078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.0.97.61:34975)
0928 05:11:27.797594 (+ 16516us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.797599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.797622 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:11:27.797676 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.797680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.797749 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.797876 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:27.801994 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.805612 (+ 3618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.807791 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.808848 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.808960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.812385 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:27.812396 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:27.812402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:27.812437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:27.817661 (+ 5224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.818876 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.818881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.818884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.819058 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:27.819476 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:27.819481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:27.819485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:27.819838 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:27.820059 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.820917 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.821135 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16390,"thread_start_us":17028,"threads_started":1}
I20260928 05:11:27.820127 6565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:27.827775 6565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:27.859362 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.847650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38161 (local address 127.0.0.1:33244)
0928 05:11:27.848356 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.848377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.848497 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.851723 (+ 3226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.851732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.851769 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:27.852204 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.852220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.854929 (+ 2709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.854938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.856794 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.856806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.858453 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.858462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.858675 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.858719 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.858881 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.859021 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":384,"threads_started":1}
I20260928 05:11:27.859385 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.847929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.0.97.2:38161)
0928 05:11:27.850907 (+ 2978us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.850919 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.850956 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:11:27.851091 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.851103 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.851453 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.852464 (+ 1011us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:27.852714 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.854721 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.857241 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.858296 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.858405 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.858609 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.858916 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.859076 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2836,"thread_start_us":2727,"threads_started":1}
I20260928 05:11:27.863639 7618 tablet_service.cc:1467] Processing CreateTablet for tablet 56bce6e5c9554a5da6ff69e40a140197 (DEFAULT_TABLE table=test-table [id=d7a394698bc74340b29c81e029408228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:27.865178 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bce6e5c9554a5da6ff69e40a140197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:27.865629 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.849692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.0.97.3:37397)
0928 05:11:27.850523 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.850571 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.857075 (+ 6504us) server_negotiation.cc:407] Connection header received
0928 05:11:27.857295 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.857301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.857549 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.857712 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:27.858587 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.860132 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.862928 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.864297 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.864429 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.864611 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.865137 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.865300 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":380,"threads_started":1}
I20260928 05:11:27.865646 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.849343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37397 (local address 127.0.0.1:52896)
0928 05:11:27.856968 (+ 7625us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.856982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.857113 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.857971 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.857975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.857993 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:27.858345 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.858357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.860320 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.860325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.862772 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.862785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.864775 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.864781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.864963 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.864999 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.865096 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.865239 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6502,"mutex_wait_us":1,"thread_start_us":5387,"threads_started":1}
I20260928 05:11:27.865701 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.848184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51650)
0928 05:11:27.856960 (+ 8776us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:27.856978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:27.858674 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:27.859452 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:27.859456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:27.859552 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:27.859795 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.859802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.861498 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.861504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.863556 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:27.863567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.864898 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.864905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:27.865101 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.865131 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:11:27.865218 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:11:27.865331 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6681,"mutex_wait_us":45,"thread_start_us":7460,"threads_started":1}
I20260928 05:11:27.866460 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:27.848444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.0.97.1:45689)
0928 05:11:27.855479 (+ 7035us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:27.855491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:27.857066 (+ 1575us) server_negotiation.cc:407] Connection header received
0928 05:11:27.858828 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:27.858836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:27.859072 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:27.859255 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:27.859959 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.861367 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.863694 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:27.864660 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:27.864807 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:27.865003 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:27.865989 (+ 986us) server_negotiation.cc:299] Negotiation successful
0928 05:11:27.866187 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6818,"thread_start_us":6703,"threads_started":1}
I20260928 05:11:27.869233 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 56bce6e5c9554a5da6ff69e40a140197 (DEFAULT_TABLE table=test-table [id=d7a394698bc74340b29c81e029408228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:27.870515 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bce6e5c9554a5da6ff69e40a140197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:27.877501 7470 tablet_service.cc:1467] Processing CreateTablet for tablet 56bce6e5c9554a5da6ff69e40a140197 (DEFAULT_TABLE table=test-table [id=d7a394698bc74340b29c81e029408228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:27.878432 7857 tablet_bootstrap.cc:492] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Bootstrap starting.
I20260928 05:11:27.879177 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bce6e5c9554a5da6ff69e40a140197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:27.882087 7857 tablet_bootstrap.cc:654] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:27.883560 7857 log.cc:824] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:27.885661 7859 tablet_bootstrap.cc:492] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Bootstrap starting.
I20260928 05:11:27.887156 7857 tablet_bootstrap.cc:492] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: No bootstrap required, opened a new log
I20260928 05:11:27.887465 7857 ts_tablet_manager.cc:1397] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 05:11:27.889402 7859 tablet_bootstrap.cc:654] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:27.890642 7859 log.cc:824] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:27.893842 7859 tablet_bootstrap.cc:492] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: No bootstrap required, opened a new log
I20260928 05:11:27.893972 7861 tablet_bootstrap.cc:492] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Bootstrap starting.
I20260928 05:11:27.894146 7859 ts_tablet_manager.cc:1397] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 05:11:27.897459 7861 tablet_bootstrap.cc:654] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:27.898108 7857 raft_consensus.cc:348] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:27.898583 7857 raft_consensus.cc:374] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:27.898690 7857 raft_consensus.cc:729] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d441a081acef4a40800dd49ad09f5f29, State: Initialized, Role: FOLLOWER
I20260928 05:11:27.898787 7861 log.cc:824] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:27.899369 7857 consensus_queue.cc:260] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:27.902000 7861 tablet_bootstrap.cc:492] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: No bootstrap required, opened a new log
I20260928 05:11:27.902331 7861 ts_tablet_manager.cc:1397] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 05:11:27.902496 7859 raft_consensus.cc:348] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:27.902917 7859 raft_consensus.cc:374] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:27.903002 7859 raft_consensus.cc:729] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c854497442c99bc3a3dfcfaa3534, State: Initialized, Role: FOLLOWER
I20260928 05:11:27.903623 7859 consensus_queue.cc:260] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:27.904594 7825 heartbeater.cc:503] Master 127.0.97.62:43605 was elected leader, sending a full tablet report...
I20260928 05:11:27.910161 7859 ts_tablet_manager.cc:1428] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260928 05:11:27.915130 7684 heartbeater.cc:503] Master 127.0.97.62:43605 was elected leader, sending a full tablet report...
I20260928 05:11:27.915745 7857 ts_tablet_manager.cc:1428] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Time spent starting tablet: real 0.028s user 0.007s sys 0.021s
I20260928 05:11:27.920364 7861 raft_consensus.cc:348] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:27.920737 7861 raft_consensus.cc:374] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:27.920830 7861 raft_consensus.cc:729] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0c6f7dbe174d28976b26e4ba2be73a, State: Initialized, Role: FOLLOWER
I20260928 05:11:27.921407 7861 consensus_queue.cc:260] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:27.922933 7861 ts_tablet_manager.cc:1428] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Time spent starting tablet: real 0.020s user 0.009s sys 0.004s
W20260928 05:11:28.001441 7827 tablet.cc:2369] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:11:28.021013 7538 tablet.cc:2369] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:11:28.065788 7863 raft_consensus.cc:482] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:28.066148 7863 raft_consensus.cc:504] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:28.068120 7863 leader_election.cc:288] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689), 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397)
I20260928 05:11:28.075541 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.068845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46973 (local address 127.0.97.3:37397)
0928 05:11:28.069076 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.069084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.069838 (+ 754us) server_negotiation.cc:407] Connection header received
0928 05:11:28.070051 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.070055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.070150 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.070283 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:28.070835 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.071553 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.073714 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.074512 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.074665 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.074748 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.075118 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.075279 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 05:11:28.075562 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.068495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41265 (local address 127.0.97.1:45689)
0928 05:11:28.068743 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.068751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.069856 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 05:11:28.069939 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.069947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.070040 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.070113 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:28.070819 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.071577 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.073856 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.074709 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.074853 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.074950 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.075189 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.075303 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:11:28.075716 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.068626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37397 (local address 127.0.97.2:46973)
0928 05:11:28.069632 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:28.069646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:28.069902 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:28.070294 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:28.070297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:28.070319 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:28.070677 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:28.070686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.071679 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.071684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:28.073250 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:28.073262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.074757 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.074766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:28.074949 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.074990 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 05:11:28.075068 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 05:11:28.075213 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":25,"thread_start_us":770,"threads_started":1}
I20260928 05:11:28.075760 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.068505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.97.2:41265)
0928 05:11:28.069626 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:28.069644 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:28.069808 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:28.070165 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:28.070171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:28.070200 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:28.070655 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:28.070668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.071745 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.071748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:28.073696 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:28.073709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.074906 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.074910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:28.075018 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.075039 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:11:28.075146 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 05:11:28.075266 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":6,"thread_start_us":466,"threads_started":1}
I20260928 05:11:28.076776 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bce6e5c9554a5da6ff69e40a140197" candidate_uuid: "d441a081acef4a40800dd49ad09f5f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" is_pre_election: true
I20260928 05:11:28.076840 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bce6e5c9554a5da6ff69e40a140197" candidate_uuid: "d441a081acef4a40800dd49ad09f5f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c854497442c99bc3a3dfcfaa3534" is_pre_election: true
I20260928 05:11:28.077267 7490 raft_consensus.cc:2454] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d441a081acef4a40800dd49ad09f5f29 in term 0.
I20260928 05:11:28.077296 7779 raft_consensus.cc:2454] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d441a081acef4a40800dd49ad09f5f29 in term 0.
I20260928 05:11:28.077920 7568 leader_election.cc:302] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b03c854497442c99bc3a3dfcfaa3534, 5e0c6f7dbe174d28976b26e4ba2be73a, d441a081acef4a40800dd49ad09f5f29; no voters:
I20260928 05:11:28.078329 7863 raft_consensus.cc:2781] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:28.078462 7863 raft_consensus.cc:482] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:28.078606 7863 raft_consensus.cc:3037] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:28.080782 7863 raft_consensus.cc:504] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:28.081321 7863 leader_election.cc:288] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [CANDIDATE]: Term 1 election: Requested vote from peers 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689), 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397)
I20260928 05:11:28.081753 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bce6e5c9554a5da6ff69e40a140197" candidate_uuid: "d441a081acef4a40800dd49ad09f5f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c854497442c99bc3a3dfcfaa3534"
I20260928 05:11:28.081763 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bce6e5c9554a5da6ff69e40a140197" candidate_uuid: "d441a081acef4a40800dd49ad09f5f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a"
I20260928 05:11:28.081924 7779 raft_consensus.cc:3037] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:28.081940 7490 raft_consensus.cc:3037] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:28.084062 7779 raft_consensus.cc:2454] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d441a081acef4a40800dd49ad09f5f29 in term 1.
I20260928 05:11:28.084272 7490 raft_consensus.cc:2454] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d441a081acef4a40800dd49ad09f5f29 in term 1.
I20260928 05:11:28.084478 7566 leader_election.cc:302] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c854497442c99bc3a3dfcfaa3534, d441a081acef4a40800dd49ad09f5f29; no voters:
I20260928 05:11:28.084795 7863 raft_consensus.cc:2781] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:28.085281 7863 raft_consensus.cc:686] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 1 LEADER]: Becoming Leader. State: Replica: d441a081acef4a40800dd49ad09f5f29, State: Running, Role: LEADER
I20260928 05:11:28.085881 7863 consensus_queue.cc:237] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } }
I20260928 05:11:28.091231 6565 catalog_manager.cc:5495] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 reported cstate change: term changed from 0 to 1, leader changed from <none> to d441a081acef4a40800dd49ad09f5f29 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d441a081acef4a40800dd49ad09f5f29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } health_report { overall_health: UNKNOWN } } }
W20260928 05:11:28.123967 7686 tablet.cc:2369] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:11:28.159061 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.153296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.0.97.2:38161)
0928 05:11:28.153504 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.153509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.153528 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:11:28.153740 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.153744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.153887 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.154058 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:28.154458 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.155288 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.156912 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.157294 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.157390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.157913 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:28.158411 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:28.158517 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.158689 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.158763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:11:28.167496 7490 raft_consensus.cc:1261] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Refusing update from remote peer d441a081acef4a40800dd49ad09f5f29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:28.167497 7779 raft_consensus.cc:1261] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Refusing update from remote peer d441a081acef4a40800dd49ad09f5f29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:28.168440 7868 consensus_queue.cc:1035] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:28.168807 7863 consensus_queue.cc:1035] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:28.206480 6565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:28.206836 6565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:28.212968 6565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540972e76935476f91b35f900d41dae7: Sending DeleteTablet for 3 replicas of tablet 56bce6e5c9554a5da6ff69e40a140197
I20260928 05:11:28.214067 7618 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bce6e5c9554a5da6ff69e40a140197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33244
I20260928 05:11:28.214164 7470 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bce6e5c9554a5da6ff69e40a140197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51650
I20260928 05:11:28.215153 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bce6e5c9554a5da6ff69e40a140197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52896
I20260928 05:11:28.215547 7880 tablet_replica.cc:330] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: stopping tablet replica
I20260928 05:11:28.216068 7880 raft_consensus.cc:2229] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:28.216883 7880 raft_consensus.cc:2258] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:28.223834 7881 tablet_replica.cc:330] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: stopping tablet replica
I20260928 05:11:28.224025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 6533
I20260928 05:11:28.224412 7881 raft_consensus.cc:2229] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:28.225039 7881 raft_consensus.cc:2258] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:28.225201 6595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:28.227582 7881 ts_tablet_manager.cc:1905] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:28.229825 7880 ts_tablet_manager.cc:1905] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:28.235461 7881 ts_tablet_manager.cc:1918] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:28.235654 7881 log.cc:1196] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/56bce6e5c9554a5da6ff69e40a140197
I20260928 05:11:28.236213 7881 ts_tablet_manager.cc:1939] T 56bce6e5c9554a5da6ff69e40a140197 P 5e0c6f7dbe174d28976b26e4ba2be73a: Deleting consensus metadata
I20260928 05:11:28.238755 7882 tablet_replica.cc:330] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: stopping tablet replica
I20260928 05:11:28.242740 7880 ts_tablet_manager.cc:1918] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:28.243311 7882 raft_consensus.cc:2229] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:28.244442 7882 raft_consensus.cc:2258] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:28.245187 7880 log.cc:1196] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/56bce6e5c9554a5da6ff69e40a140197
I20260928 05:11:28.246699 7882 ts_tablet_manager.cc:1905] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:28.250547 7880 ts_tablet_manager.cc:1939] T 56bce6e5c9554a5da6ff69e40a140197 P d441a081acef4a40800dd49ad09f5f29: Deleting consensus metadata
I20260928 05:11:28.256995 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.218921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.97.61:34975)
0928 05:11:28.219121 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.219128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.223179 (+ 4051us) server_negotiation.cc:407] Connection header received
0928 05:11:28.223381 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.223386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.223471 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.223588 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:28.238685 (+ 15097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.239871 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.241764 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.242513 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.242679 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.246708 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:28.246717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:28.246724 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:28.246790 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:28.250464 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.251360 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.251366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.251370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.251510 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.251927 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.251933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.251937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.252281 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:28.252512 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.256392 (+ 3880us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.256641 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 05:11:28.257467 7882 ts_tablet_manager.cc:1918] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:28.257705 7882 log.cc:1196] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/56bce6e5c9554a5da6ff69e40a140197
I20260928 05:11:28.258355 7882 ts_tablet_manager.cc:1939] T 56bce6e5c9554a5da6ff69e40a140197 P 1b03c854497442c99bc3a3dfcfaa3534: Deleting consensus metadata
I20260928 05:11:28.273065 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.223301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.0.97.60:45951)
0928 05:11:28.223484 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.223490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.253940 (+ 30450us) server_negotiation.cc:407] Connection header received
0928 05:11:28.254240 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.254247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.254331 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.254400 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:28.258487 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.259647 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.261678 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.262327 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.262428 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.266076 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:28.266083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:28.266088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:28.266128 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:28.269831 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.270666 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.270672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.270676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.270825 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.271260 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.271265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.271269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.271613 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:28.271829 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.272311 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.272544 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:11:28.356837 6551 catalog_manager.cc:4841] TS 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689): tablet 56bce6e5c9554a5da6ff69e40a140197 (table test-table [id=d7a394698bc74340b29c81e029408228]) successfully deleted
I20260928 05:11:28.358103 6550 catalog_manager.cc:4841] TS 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397): tablet 56bce6e5c9554a5da6ff69e40a140197 (table test-table [id=d7a394698bc74340b29c81e029408228]) successfully deleted
I20260928 05:11:28.358395 6550 catalog_manager.cc:4841] TS d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161): tablet 56bce6e5c9554a5da6ff69e40a140197 (table test-table [id=d7a394698bc74340b29c81e029408228]) successfully deleted
I20260928 05:11:28.358803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 6533
I20260928 05:11:28.402930 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.384622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.0.97.61:34975)
0928 05:11:28.384846 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.384854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.385785 (+ 931us) server_negotiation.cc:407] Connection header received
0928 05:11:28.386095 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.386104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.386219 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.386320 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:28.389381 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.390983 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.393207 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.393610 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.393747 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.396909 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:28.396915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:28.396919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:28.396957 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:28.399919 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.400723 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.400728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.400732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.400850 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.401274 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.401279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.401284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.401663 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:28.401910 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.402303 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.402557 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:11:28.408432 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:28.385520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.0.97.60:45951)
0928 05:11:28.385728 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:28.385734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:28.392019 (+ 6285us) server_negotiation.cc:407] Connection header received
0928 05:11:28.392556 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:28.392562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:28.392647 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:28.392750 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:28.394911 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.396134 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.398017 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:28.398763 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:28.398896 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:28.403029 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:28.403036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:28.403040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:28.403078 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:28.405794 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.406488 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.406493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.406496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.406660 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:28.407008 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:28.407012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:28.407016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:28.407359 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:28.407612 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:28.407903 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 05:11:28.408108 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:11:29.188799 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.187846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:34583)
0928 05:11:29.188418 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.188547 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":323,"threads_started":1}
W20260928 05:11:29.189224 7536 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43605 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.195082 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.194244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:58047)
0928 05:11:29.194801 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.194916 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":214,"threads_started":1}
W20260928 05:11:29.195493 7825 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43605 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.201598 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.200929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:51165)
0928 05:11:29.201323 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.201443 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":194,"threads_started":1}
W20260928 05:11:29.201980 7684 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43605 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.713542 7895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 540972e76935476f91b35f900d41dae7)
I20260928 05:11:29.713747 7895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } }
I20260928 05:11:29.714232 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 540972e76935476f91b35f900d41dae7)
I20260928 05:11:29.714435 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } }
I20260928 05:11:29.714897 7895 leader_election.cc:288] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605), 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61:34975)
I20260928 05:11:29.715346 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605), 2295905206ad47ac84aae1ef0b8739db (127.0.97.60:45951)
I20260928 05:11:29.716511 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.715038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:41944)
0928 05:11:29.715864 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.716170 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":1001,"threads_started":1}
W20260928 05:11:29.716873 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.717065 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.715885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:41946)
0928 05:11:29.716620 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.716797 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":449,"threads_started":1}
I20260928 05:11:29.718793 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.718371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:41948)
0928 05:11:29.718501 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.718626 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:11:29.719033 7339 leader_election.cc:334] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
W20260928 05:11:29.719822 7148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.722431 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.721839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:41964)
0928 05:11:29.722017 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.722205 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 05:11:29.722689 7148 leader_election.cc:334] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.724113 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.715270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.0.97.61:34975)
0928 05:11:29.715878 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.715884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.716510 (+ 626us) server_negotiation.cc:407] Connection header received
0928 05:11:29.716811 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.716816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.716900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.716980 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.717890 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.719502 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.722034 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.723115 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.723272 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.723362 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.723643 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.723769 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":501,"thread_start_us":345,"threads_started":1}
I20260928 05:11:29.724218 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.715313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34975 (local address 127.0.0.1:52198)
0928 05:11:29.716405 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.716422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.716619 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.717205 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.717213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.717245 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.717619 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.717646 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.719706 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.719711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.721907 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.721921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.723299 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.723305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.723475 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.723501 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.723631 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.723771 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":4,"thread_start_us":282,"threads_started":1}
I20260928 05:11:29.724748 7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2295905206ad47ac84aae1ef0b8739db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50b19a82457942dc91b0ee156ee92f21" is_pre_election: true
I20260928 05:11:29.725020 7174 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2295905206ad47ac84aae1ef0b8739db in term 1.
I20260928 05:11:29.725261 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.717101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.97.60:45951)
0928 05:11:29.718940 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.718948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.718966 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:11:29.719029 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.719034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.719107 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.719202 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.719871 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.720800 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.723274 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.724137 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.724321 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.724429 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.724835 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.724957 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1706,"thread_start_us":1556,"threads_started":1}
I20260928 05:11:29.725327 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.716657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45951 (local address 127.0.0.1:47392)
0928 05:11:29.717255 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.717275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.717393 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.719331 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.719336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.719367 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.719719 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.719732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.720925 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.720932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.723109 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.723120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.724546 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.724551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.724672 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.724695 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.724813 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.724949 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":52,"thread_start_us":2878,"threads_started":1}
I20260928 05:11:29.725498 7342 leader_election.cc:302] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2295905206ad47ac84aae1ef0b8739db, 50b19a82457942dc91b0ee156ee92f21; no voters: 540972e76935476f91b35f900d41dae7
I20260928 05:11:29.725924 7895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 05:11:29.725879 7365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50b19a82457942dc91b0ee156ee92f21" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2295905206ad47ac84aae1ef0b8739db" is_pre_election: true
I20260928 05:11:29.726039 7895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 540972e76935476f91b35f900d41dae7)
I20260928 05:11:29.726138 7895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:11:29.728322 7895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } }
I20260928 05:11:29.728729 7365 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50b19a82457942dc91b0ee156ee92f21 in current term 2: Already voted for candidate 2295905206ad47ac84aae1ef0b8739db in this term.
I20260928 05:11:29.728981 7895 leader_election.cc:288] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 2 election: Requested vote from peers 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605), 50b19a82457942dc91b0ee156ee92f21 (127.0.97.61:34975)
I20260928 05:11:29.729461 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.729088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:41976)
0928 05:11:29.729243 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.729334 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:11:29.729475 7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2295905206ad47ac84aae1ef0b8739db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50b19a82457942dc91b0ee156ee92f21"
I20260928 05:11:29.729712 7174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:11:29.730333 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.730054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:41992)
0928 05:11:29.730167 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.730236 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 05:11:29.730561 7339 leader_election.cc:334] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
I20260928 05:11:29.730863 7148 leader_election.cc:302] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50b19a82457942dc91b0ee156ee92f21; no voters: 2295905206ad47ac84aae1ef0b8739db, 540972e76935476f91b35f900d41dae7
I20260928 05:11:29.732190 7174 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2295905206ad47ac84aae1ef0b8739db in term 2.
I20260928 05:11:29.732380 7896 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:11:29.732595 7342 leader_election.cc:302] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2295905206ad47ac84aae1ef0b8739db, 50b19a82457942dc91b0ee156ee92f21; no voters: 540972e76935476f91b35f900d41dae7
I20260928 05:11:29.732853 7895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:11:29.733373 7895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [term 2 LEADER]: Becoming Leader. State: Replica: 2295905206ad47ac84aae1ef0b8739db, State: Running, Role: LEADER
I20260928 05:11:29.733824 7895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } }
I20260928 05:11:29.735874 7905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2295905206ad47ac84aae1ef0b8739db. Latest consensus state: current_term: 2 leader_uuid: "2295905206ad47ac84aae1ef0b8739db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:29.736292 7905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:29.736897 7907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:29.740543 7907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:29.741482 7907 catalog_manager.cc:1259] Loaded cluster ID: 3a2752c27580474ca021d7bf60aa5459
I20260928 05:11:29.741580 7907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:29.742885 7907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:29.743650 7907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Loaded TSK: 0
I20260928 05:11:29.744609 7907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:11:29.799690 7354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:29.806734 7354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:29.811014 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.810638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:42028)
0928 05:11:29.810798 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.810890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:11:29.811066 7174 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [term 2 FOLLOWER]: Refusing update from remote peer 2295905206ad47ac84aae1ef0b8739db: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 05:11:29.811836 7905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 05:11:29.812103 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.811713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:42044)
0928 05:11:29.811881 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.811951 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:11:29.812258 7339 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db -> Peer 540972e76935476f91b35f900d41dae7 (127.0.97.62:43605): Couldn't send request to peer 540972e76935476f91b35f900d41dae7. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 05:11:29.814394 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2295905206ad47ac84aae1ef0b8739db. Latest consensus state: current_term: 2 leader_uuid: "2295905206ad47ac84aae1ef0b8739db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:29.814678 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:29.816241 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2295905206ad47ac84aae1ef0b8739db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:29.816512 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2295905206ad47ac84aae1ef0b8739db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "540972e76935476f91b35f900d41dae7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43605 } } peers { permanent_uuid: "50b19a82457942dc91b0ee156ee92f21" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34975 } attrs { promote: false } } peers { permanent_uuid: "2295905206ad47ac84aae1ef0b8739db" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45951 } attrs { promote: false } } }
I20260928 05:11:29.816668 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:29.816764 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b19a82457942dc91b0ee156ee92f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:11:29.831799 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.824057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37397 (local address 127.0.0.1:52902)
0928 05:11:29.824199 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.824219 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.824326 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.825793 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.825798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.825831 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.826055 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.826067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.827848 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.827856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.829639 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.829652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.831072 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.831081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.831275 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.831297 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.831398 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.831523 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":46}
I20260928 05:11:29.831822 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.824971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.0.97.3:37397)
0928 05:11:29.825415 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.825421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.825438 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:11:29.825518 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.825523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.825613 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.826199 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.826364 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.827736 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.829799 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.830908 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.831047 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.831152 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.831420 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.831544 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260928 05:11:29.832644 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 14591f350d0147998a79c7eb6f263e31 (DEFAULT_TABLE table=test-table [id=84589602ab254c398260aa71869bbb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:29.832870 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.825446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38161 (local address 127.0.0.1:33270)
0928 05:11:29.826037 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.826058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.826170 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.826715 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.826721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.826746 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.827110 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.827121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.828511 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.828521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.830455 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.830469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.832107 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.832123 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.832283 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.832315 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.832392 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.832516 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":24,"thread_start_us":286,"threads_started":1}
I20260928 05:11:29.833105 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14591f350d0147998a79c7eb6f263e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:29.833664 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.824596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51654)
0928 05:11:29.824722 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.824735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.824811 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.826946 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.826950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.826980 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.827330 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.827342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.828735 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.828741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.831097 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.831110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.833023 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.833030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.833197 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.833227 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.833287 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.833403 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":304}
I20260928 05:11:29.833665 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.825842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.97.2:38161)
0928 05:11:29.826279 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.826285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.826301 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:11:29.826364 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.826368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.826442 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.826563 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.827296 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.828374 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.830630 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.831783 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.833053 (+ 1270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.833170 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.833219 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.833354 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":520,"threads_started":1}
I20260928 05:11:29.834684 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.824605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.97.1:45689)
0928 05:11:29.825554 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.825558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.825573 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:11:29.825658 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.825665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.826785 (+ 1120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.827471 (+ 686us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.827614 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.828617 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.831284 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.832803 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.834108 (+ 1305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.834208 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.834258 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.834383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":752,"threads_started":1}
I20260928 05:11:29.835289 7470 tablet_service.cc:1467] Processing CreateTablet for tablet 14591f350d0147998a79c7eb6f263e31 (DEFAULT_TABLE table=test-table [id=84589602ab254c398260aa71869bbb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:29.835655 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14591f350d0147998a79c7eb6f263e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:29.836957 7618 tablet_service.cc:1467] Processing CreateTablet for tablet 14591f350d0147998a79c7eb6f263e31 (DEFAULT_TABLE table=test-table [id=84589602ab254c398260aa71869bbb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:29.837386 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14591f350d0147998a79c7eb6f263e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:29.839003 7918 tablet_bootstrap.cc:492] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Bootstrap starting.
I20260928 05:11:29.840584 7918 tablet_bootstrap.cc:654] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:29.842366 7918 tablet_bootstrap.cc:492] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: No bootstrap required, opened a new log
I20260928 05:11:29.842518 7918 ts_tablet_manager.cc:1397] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 05:11:29.843048 7920 tablet_bootstrap.cc:492] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Bootstrap starting.
I20260928 05:11:29.843397 7918 raft_consensus.cc:348] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.843631 7918 raft_consensus.cc:374] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:29.843770 7918 raft_consensus.cc:729] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c854497442c99bc3a3dfcfaa3534, State: Initialized, Role: FOLLOWER
I20260928 05:11:29.844031 7918 consensus_queue.cc:260] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.845139 7920 tablet_bootstrap.cc:654] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:29.846544 7918 ts_tablet_manager.cc:1428] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 05:11:29.851745 7824 heartbeater.cc:503] Master 127.0.97.60:45951 was elected leader, sending a full tablet report...
I20260928 05:11:29.851919 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.847215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:54821)
0928 05:11:29.851521 (+ 4306us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.851642 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4161,"thread_start_us":4049,"threads_started":1}
I20260928 05:11:29.853128 7921 tablet_bootstrap.cc:492] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Bootstrap starting.
I20260928 05:11:29.853587 7920 tablet_bootstrap.cc:492] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: No bootstrap required, opened a new log
I20260928 05:11:29.853798 7920 ts_tablet_manager.cc:1397] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260928 05:11:29.854034 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.852423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:49717)
0928 05:11:29.853817 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.853901 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1291,"threads_started":1}
I20260928 05:11:29.854723 7920 raft_consensus.cc:348] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.854986 7920 raft_consensus.cc:374] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:29.855032 7921 tablet_bootstrap.cc:654] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:29.855088 7920 raft_consensus.cc:729] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0c6f7dbe174d28976b26e4ba2be73a, State: Initialized, Role: FOLLOWER
I20260928 05:11:29.855360 7920 consensus_queue.cc:260] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.856284 7920 ts_tablet_manager.cc:1428] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:29.863525 7535 heartbeater.cc:503] Master 127.0.97.60:45951 was elected leader, sending a full tablet report...
I20260928 05:11:29.863719 7921 tablet_bootstrap.cc:492] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: No bootstrap required, opened a new log
I20260928 05:11:29.863848 7921 ts_tablet_manager.cc:1397] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260928 05:11:29.863852 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.857046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:47217)
0928 05:11:29.863387 (+ 6341us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.863542 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6199,"thread_start_us":6045,"threads_started":1}
I20260928 05:11:29.864634 7921 raft_consensus.cc:348] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.864894 7921 raft_consensus.cc:374] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:29.865002 7921 raft_consensus.cc:729] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d441a081acef4a40800dd49ad09f5f29, State: Initialized, Role: FOLLOWER
I20260928 05:11:29.865036 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.864691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:56551)
0928 05:11:29.864835 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.864911 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:11:29.865284 7921 consensus_queue.cc:260] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.866164 7921 ts_tablet_manager.cc:1428] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260928 05:11:29.867322 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.866371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:39541)
0928 05:11:29.866939 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.867061 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":313,"threads_started":1}
I20260928 05:11:29.868523 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.868161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:39555)
0928 05:11:29.868303 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.868393 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 05:11:29.868541 7683 heartbeater.cc:503] Master 127.0.97.60:45951 was elected leader, sending a full tablet report...
I20260928 05:11:29.918743 7922 raft_consensus.cc:482] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:29.919008 7922 raft_consensus.cc:504] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.921010 7922 leader_election.cc:288] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689), d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161)
I20260928 05:11:29.929036 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.921525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47921 (local address 127.0.97.1:45689)
0928 05:11:29.921777 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.921783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.921811 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:11:29.921998 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.922003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.922076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.922177 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.922775 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.923844 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.926691 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.927888 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.928032 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.928135 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.928617 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.928778 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 05:11:29.929036 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.921475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34421 (local address 127.0.97.2:38161)
0928 05:11:29.921689 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.921696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.921805 (+ 109us) server_negotiation.cc:407] Connection header received
0928 05:11:29.921999 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.922004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.922076 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.922183 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:29.922808 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.923829 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.926676 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.927915 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.928038 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.928135 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.928616 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.928767 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:11:29.929141 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.921258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.97.3:47921)
0928 05:11:29.921651 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.921671 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.921822 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.922207 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.922212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.922259 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.922615 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.922632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.923991 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.923997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.926469 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.926485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.928119 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.928125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.928398 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.928462 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.928558 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.928741 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":247,"spinlock_wait_cycles":8960}
I20260928 05:11:29.929169 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.921258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38161 (local address 127.0.97.3:34421)
0928 05:11:29.921715 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.921731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:29.921887 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:29.922207 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:29.922212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:29.922258 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:29.922661 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.922672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.923991 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.923997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.926469 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:29.926485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.928119 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.928125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:29.928402 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.928458 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:11:29.928558 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 05:11:29.928741 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":341,"spinlock_wait_cycles":3712}
I20260928 05:11:29.930172 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14591f350d0147998a79c7eb6f263e31" candidate_uuid: "1b03c854497442c99bc3a3dfcfaa3534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" is_pre_election: true
I20260928 05:11:29.930137 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14591f350d0147998a79c7eb6f263e31" candidate_uuid: "1b03c854497442c99bc3a3dfcfaa3534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d441a081acef4a40800dd49ad09f5f29" is_pre_election: true
I20260928 05:11:29.930419 7490 raft_consensus.cc:2454] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c854497442c99bc3a3dfcfaa3534 in term 0.
I20260928 05:11:29.930510 7638 raft_consensus.cc:2454] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c854497442c99bc3a3dfcfaa3534 in term 0.
I20260928 05:11:29.931007 7714 leader_election.cc:302] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c854497442c99bc3a3dfcfaa3534, 5e0c6f7dbe174d28976b26e4ba2be73a; no voters:
I20260928 05:11:29.931397 7922 raft_consensus.cc:2781] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:29.931517 7922 raft_consensus.cc:482] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:29.931618 7922 raft_consensus.cc:3037] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:29.933640 7922 raft_consensus.cc:504] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.934239 7922 leader_election.cc:288] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [CANDIDATE]: Term 1 election: Requested vote from peers 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689), d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161)
I20260928 05:11:29.934674 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14591f350d0147998a79c7eb6f263e31" candidate_uuid: "1b03c854497442c99bc3a3dfcfaa3534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a"
I20260928 05:11:29.934758 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14591f350d0147998a79c7eb6f263e31" candidate_uuid: "1b03c854497442c99bc3a3dfcfaa3534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d441a081acef4a40800dd49ad09f5f29"
I20260928 05:11:29.934870 7490 raft_consensus.cc:3037] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:29.934988 7638 raft_consensus.cc:3037] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:29.937002 7490 raft_consensus.cc:2454] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c854497442c99bc3a3dfcfaa3534 in term 1.
I20260928 05:11:29.937151 7638 raft_consensus.cc:2454] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c854497442c99bc3a3dfcfaa3534 in term 1.
I20260928 05:11:29.937378 7714 leader_election.cc:302] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c854497442c99bc3a3dfcfaa3534, 5e0c6f7dbe174d28976b26e4ba2be73a; no voters:
I20260928 05:11:29.937680 7922 raft_consensus.cc:2781] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:29.938184 7922 raft_consensus.cc:686] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 LEADER]: Becoming Leader. State: Replica: 1b03c854497442c99bc3a3dfcfaa3534, State: Running, Role: LEADER
I20260928 05:11:29.938773 7922 consensus_queue.cc:237] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } }
I20260928 05:11:29.939313 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.938833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:41271)
0928 05:11:29.939074 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.939164 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 05:11:29.942402 7354 catalog_manager.cc:5495] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1b03c854497442c99bc3a3dfcfaa3534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c854497442c99bc3a3dfcfaa3534" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 } health_report { overall_health: UNKNOWN } } }
I20260928 05:11:29.987437 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.982124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.0.97.3:37397)
0928 05:11:29.982310 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:29.982314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:29.982330 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:11:29.982412 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:29.982415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:29.982511 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:29.982681 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:29.983061 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.983774 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.985058 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:29.985552 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:29.985666 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:29.986060 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:29.986762 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:29.986889 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:29.987050 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 05:11:29.987116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:11:29.991703 7490 raft_consensus.cc:1261] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Refusing update from remote peer 1b03c854497442c99bc3a3dfcfaa3534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:29.992013 7638 raft_consensus.cc:1261] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Refusing update from remote peer 1b03c854497442c99bc3a3dfcfaa3534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:29.992332 7931 consensus_queue.cc:1035] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e0c6f7dbe174d28976b26e4ba2be73a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:29.992748 7922 consensus_queue.cc:1035] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d441a081acef4a40800dd49ad09f5f29" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:29.994910 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.994280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:37771)
0928 05:11:29.994567 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.994693 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 05:11:29.996079 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:29.995583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:59215)
0928 05:11:29.995818 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:29.995928 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 05:11:30.001709 7936 mvcc.cc:205] Tried to move back new op lower bound from 7334184099799539712 to 7334184099593179136. Current Snapshot: MvccSnapshot[applied={T|T < 7334184099799539712}]
I20260928 05:11:30.002471 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.002005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:38429)
0928 05:11:30.002216 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.002303 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 05:11:30.004208 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.002615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.2:57931)
0928 05:11:30.003958 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.004044 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1232}
I20260928 05:11:30.008635 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.008263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.3:53537)
0928 05:11:30.008382 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.008492 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 05:11:30.010504 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.010077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.97.1:39211)
0928 05:11:30.010274 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.010356 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 05:11:30.015170 7935 mvcc.cc:205] Tried to move back new op lower bound from 7334184099799539712 to 7334184099593179136. Current Snapshot: MvccSnapshot[applied={T|T < 7334184099799539712 or (T in {7334184099799539712})}]
I20260928 05:11:30.015908 7354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:30.016161 7354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:30.022177 7354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2295905206ad47ac84aae1ef0b8739db: Sending DeleteTablet for 3 replicas of tablet 14591f350d0147998a79c7eb6f263e31
I20260928 05:11:30.023000 7470 tablet_service.cc:1514] Processing DeleteTablet for tablet 14591f350d0147998a79c7eb6f263e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:30 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51654
I20260928 05:11:30.023748 7469 tablet_service.cc:1514] Processing DeleteTablet for tablet 14591f350d0147998a79c7eb6f263e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:30 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51654
I20260928 05:11:30.024080 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet 14591f350d0147998a79c7eb6f263e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:30 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52902
I20260928 05:11:30.024307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 7333
I20260928 05:11:30.024484 7618 tablet_service.cc:1514] Processing DeleteTablet for tablet 14591f350d0147998a79c7eb6f263e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:30 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33270
I20260928 05:11:30.030975 7945 tablet_replica.cc:330] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: stopping tablet replica
I20260928 05:11:30.031234 7945 raft_consensus.cc:2229] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:30.031574 7945 raft_consensus.cc:2258] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:30.033371 7945 ts_tablet_manager.cc:1905] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:30.036387 7943 tablet_replica.cc:330] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: stopping tablet replica
I20260928 05:11:30.036662 7943 raft_consensus.cc:2229] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:30.038621 7944 tablet_replica.cc:330] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: stopping tablet replica
I20260928 05:11:30.038828 7944 raft_consensus.cc:2229] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:30.039461 7944 raft_consensus.cc:2258] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:30.039964 7943 raft_consensus.cc:2258] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:30.040575 7944 ts_tablet_manager.cc:1905] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:30.041666 7943 ts_tablet_manager.cc:1905] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:30.047778 7944 ts_tablet_manager.cc:1918] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:30.047926 7944 log.cc:1196] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/14591f350d0147998a79c7eb6f263e31
I20260928 05:11:30.048406 7944 ts_tablet_manager.cc:1939] T 14591f350d0147998a79c7eb6f263e31 P 1b03c854497442c99bc3a3dfcfaa3534: Deleting consensus metadata
I20260928 05:11:30.049577 7340 catalog_manager.cc:4841] TS 1b03c854497442c99bc3a3dfcfaa3534 (127.0.97.3:37397): tablet 14591f350d0147998a79c7eb6f263e31 (table test-table [id=84589602ab254c398260aa71869bbb6e]) successfully deleted
I20260928 05:11:30.049633 7945 ts_tablet_manager.cc:1918] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:30.049760 7945 log.cc:1196] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/14591f350d0147998a79c7eb6f263e31
I20260928 05:11:30.050258 7945 ts_tablet_manager.cc:1939] T 14591f350d0147998a79c7eb6f263e31 P d441a081acef4a40800dd49ad09f5f29: Deleting consensus metadata
I20260928 05:11:30.051385 7340 catalog_manager.cc:4841] TS d441a081acef4a40800dd49ad09f5f29 (127.0.97.2:38161): tablet 14591f350d0147998a79c7eb6f263e31 (table test-table [id=84589602ab254c398260aa71869bbb6e]) successfully deleted
I20260928 05:11:30.054661 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.025776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.97.1:45689)
0928 05:11:30.025931 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:30.025937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:30.036687 (+ 10750us) server_negotiation.cc:407] Connection header received
0928 05:11:30.036910 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:30.036916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:30.036996 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:30.037125 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:30.038660 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:30.039623 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:30.042015 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:30.042751 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:30.042861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:30.046710 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:30.046731 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:30.046754 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:30.046804 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:30.051202 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:30.052150 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:30.052169 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:30.052174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:30.052303 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:30.052755 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:30.052761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:30.052765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:30.053271 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:30.053606 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:30.053928 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 05:11:30.054177 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:11:30.055486 7530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:30.174206 7943 ts_tablet_manager.cc:1918] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:30.174434 7943 log.cc:1196] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/14591f350d0147998a79c7eb6f263e31
I20260928 05:11:30.174877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 7333
I20260928 05:11:30.174971 7943 ts_tablet_manager.cc:1939] T 14591f350d0147998a79c7eb6f263e31 P 5e0c6f7dbe174d28976b26e4ba2be73a: Deleting consensus metadata
I20260928 05:11:30.193275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 7547
I20260928 05:11:30.193650 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.193174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51674)
0928 05:11:30.193368 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.193456 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:11:30.193922 7341 catalog_manager.cc:4584] TS 5e0c6f7dbe174d28976b26e4ba2be73a (127.0.97.1:45689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14591f350d0147998a79c7eb6f263e31: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
I20260928 05:11:30.208770 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.194172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.97.2:38161)
0928 05:11:30.194338 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:30.194349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:30.194612 (+ 263us) server_negotiation.cc:407] Connection header received
0928 05:11:30.194811 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:30.194815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:30.194877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:30.194957 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:30.196688 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:30.197419 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:30.199296 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:30.199695 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:30.199809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:30.202930 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:30.202945 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:30.202962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:30.203000 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:30.205773 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:30.206631 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:30.206642 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:30.206649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:30.206800 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:30.207213 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:30.207217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:30.207221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:30.207622 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:30.207929 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:30.208209 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 05:11:30.208450 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:11:30.209407 7678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:30.246409 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.245929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51686)
0928 05:11:30.246140 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.246240 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:11:30.247537 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.247164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51690)
0928 05:11:30.247328 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.247406 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 05:11:30.248522 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.248188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51704)
0928 05:11:30.248326 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.248396 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:11:30.323011 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.322465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51714)
0928 05:11:30.322719 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.322841 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 05:11:30.324182 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.323741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51728)
0928 05:11:30.323919 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.324011 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:11:30.329672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 7547
I20260928 05:11:30.331502 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.331013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51744)
0928 05:11:30.331251 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.331339 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 05:11:30.332659 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.332261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51758)
0928 05:11:30.332414 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.332501 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:11:30.340364 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.339900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:42058)
0928 05:11:30.340098 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.340197 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 05:11:30.341441 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.341016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43605 (local address 127.0.0.1:42062)
0928 05:11:30.341190 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.341283 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:11:30.352782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 7696
I20260928 05:11:30.370952 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.353552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.97.3:37397)
0928 05:11:30.353720 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:30.353724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:30.353806 (+ 82us) server_negotiation.cc:407] Connection header received
0928 05:11:30.353988 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:30.353991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:30.354051 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:30.354130 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:30.356055 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:30.357267 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:30.359115 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:30.359585 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:30.359691 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:30.363018 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:30.363035 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:30.363056 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:30.363094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:30.367108 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:30.367929 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:30.367940 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:30.367943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:30.368120 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:30.368577 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:30.368582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:30.368586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:30.369012 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:30.369315 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:30.370345 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0928 05:11:30.370587 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:11:30.371830 7819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:30.410243 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.409748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51766)
0928 05:11:30.409972 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.410076 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:11:30.411329 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.410930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51774)
0928 05:11:30.411081 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.411177 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:11:30.497537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 7696
I20260928 05:11:30.519450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 7026
I20260928 05:11:30.520604 7194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:30.568109 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.567625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51776)
0928 05:11:30.567815 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.567931 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 05:11:30.569072 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.568716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51780)
0928 05:11:30.568864 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.568943 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:11:30.581233 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.580798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51782)
0928 05:11:30.580985 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.581081 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 05:11:30.582185 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:30.581801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45689 (local address 127.0.0.1:51786)
0928 05:11:30.581947 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:30.582032 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 05:11:30.636445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 7026
I20260928 05:11:30.657092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 7127
I20260928 05:11:30.658217 7385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:30.777457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 7127
2026-09-28T05:11:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9594 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:30 dist-test-slave-6718 krb5kdc[7954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:30 dist-test-slave-6718 krb5kdc[7954](info): set up 2 sockets
Sep 28 05:11:30 dist-test-slave-6718 krb5kdc[7954](info): commencing operation
krb5kdc: starting...
W20260928 05:11:31.875523 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:32 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 05:11:32.082695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34727
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:32.397883 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:32.398187 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:32.398347 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:32.405579 7967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:32.405687 7967 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 05:11:32.405751 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:32.405839 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:32.405898 7967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:32.405972 7967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:32.415761 7967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34727
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:32.417173 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:32.418586 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:32.430090 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:32.430130 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:32.430276 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:32.431105 7967 server_base.cc:1034] running on GCE node
I20260928 05:11:32.431819 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:32.433071 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:32.434293 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790572292434216 us; error 48 us; skew 500 ppm
I20260928 05:11:32.434667 7967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:32 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:32.437655 7967 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 05:11:32.439024 7967 webserver.cc:460] Webserver started at http://127.0.97.62:44629/ using document root <none> and password file <none>
I20260928 05:11:32.439674 7967 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:32.439790 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:32.440110 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:32.442919 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "60584990d27348af81b442b71dcd92b0"
format_stamp: "Formatted at 2026-09-28 05:11:32 on dist-test-slave-6718"
server_key: "7a2144c637f725715544b49c79a224b4"
server_key_iv: "55d95ecfd3e57e317861fc03a0c9b7af"
server_key_version: "encryptionkey@0"
I20260928 05:11:32.443681 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "60584990d27348af81b442b71dcd92b0"
format_stamp: "Formatted at 2026-09-28 05:11:32 on dist-test-slave-6718"
server_key: "7a2144c637f725715544b49c79a224b4"
server_key_iv: "55d95ecfd3e57e317861fc03a0c9b7af"
server_key_version: "encryptionkey@0"
I20260928 05:11:32.448372 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 05:11:32.451546 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:32.452807 7967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:11:32.452980 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "60584990d27348af81b442b71dcd92b0"
format_stamp: "Formatted at 2026-09-28 05:11:32 on dist-test-slave-6718"
server_key: "7a2144c637f725715544b49c79a224b4"
server_key_iv: "55d95ecfd3e57e317861fc03a0c9b7af"
server_key_version: "encryptionkey@0"
I20260928 05:11:32.453199 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:32.471534 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:32.475653 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:32.475907 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:32.496549 7967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34727
I20260928 05:11:32.496557 8043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34727 every 8 connection(s)
I20260928 05:11:32.497859 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:32.501097 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 7967
I20260928 05:11:32.501325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:32.502077 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:32.502164 388 external_mini_cluster.cc:1428] Setting key 500b6eec1ddd0f5b7f6e9eb653880e9e
I20260928 05:11:32.514295 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Bootstrap starting.
Sep 28 05:11:32 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:32.517274 8044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:32.518420 8044 log.cc:824] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:32.521081 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: No bootstrap required, opened a new log
I20260928 05:11:32.523468 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:32.504248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.0.97.62:34727)
0928 05:11:32.505005 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:32.505016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:32.505058 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:11:32.506973 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:32.506994 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:32.507383 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:32.507794 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:32.509163 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:32.510380 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:32.511608 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:32.512322 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:32.512537 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:32.516304 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:32.516311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:32.516324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:32.516362 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:32.519495 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:32.520239 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:32.520247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:32.520253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:32.520417 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:32.520785 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:32.520794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:32.520797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:32.521340 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:32.521646 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:32.522048 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 05:11:32.522456 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":325,"threads_started":1}
I20260928 05:11:32.528810 8044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60584990d27348af81b442b71dcd92b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34727 } }
I20260928 05:11:32.529242 8044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:32.529369 8044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60584990d27348af81b442b71dcd92b0, State: Initialized, Role: FOLLOWER
I20260928 05:11:32.529889 8044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60584990d27348af81b442b71dcd92b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34727 } }
I20260928 05:11:32.530112 8044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:32.530257 8044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:32.530413 8044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:32.532424 8044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60584990d27348af81b442b71dcd92b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34727 } }
I20260928 05:11:32.532896 8044 leader_election.cc:302] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60584990d27348af81b442b71dcd92b0; no voters:
I20260928 05:11:32.533434 8044 leader_election.cc:288] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:32.533584 8049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:32.534262 8049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [term 1 LEADER]: Becoming Leader. State: Replica: 60584990d27348af81b442b71dcd92b0, State: Running, Role: LEADER
I20260928 05:11:32.534857 8049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60584990d27348af81b442b71dcd92b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34727 } }
I20260928 05:11:32.535456 8044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:32.537926 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60584990d27348af81b442b71dcd92b0. Latest consensus state: current_term: 1 leader_uuid: "60584990d27348af81b442b71dcd92b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60584990d27348af81b442b71dcd92b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34727 } } }
I20260928 05:11:32.537896 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60584990d27348af81b442b71dcd92b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60584990d27348af81b442b71dcd92b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34727 } } }
I20260928 05:11:32.538339 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:32.538357 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:32.540320 8058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:32.545894 8058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:32.553988 8058 catalog_manager.cc:1347] Generated new cluster ID: df569c1be0ee43ca95d143924cc28bb1
I20260928 05:11:32.554101 8058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:32.586099 8058 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:32.587463 8058 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:32.595593 8058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Generated new TSK 0
I20260928 05:11:32.596413 8058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 05:11:32.654510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34727
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:32.975453 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:32.975744 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:32.975909 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:32.982758 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:32.982887 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:32.983026 8070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:11:32.983160 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:32.992370 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:32.993552 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:32.994923 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:33.007404 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:33.007426 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:33.007437 8078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:33.008455 8070 server_base.cc:1034] running on GCE node
I20260928 05:11:33.009034 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:33.009924 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:33.011157 8070 hybrid_clock.cc:648] HybridClock initialized: now 1790572293011085 us; error 48 us; skew 500 ppm
I20260928 05:11:33.011472 8070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:33 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:33.014582 8070 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:11:33.016031 8070 webserver.cc:460] Webserver started at http://127.0.97.1:46387/ using document root <none> and password file <none>
I20260928 05:11:33.016722 8070 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:33.016842 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:33.017148 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:33.020018 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "372ebd49ce994682a3e179a6f9c90f3d"
format_stamp: "Formatted at 2026-09-28 05:11:33 on dist-test-slave-6718"
server_key: "c0dd801e0000795f607cf4d20dd9eac8"
server_key_iv: "714d05f398730ea9b7e6784a0a525549"
server_key_version: "encryptionkey@0"
I20260928 05:11:33.020789 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "372ebd49ce994682a3e179a6f9c90f3d"
format_stamp: "Formatted at 2026-09-28 05:11:33 on dist-test-slave-6718"
server_key: "c0dd801e0000795f607cf4d20dd9eac8"
server_key_iv: "714d05f398730ea9b7e6784a0a525549"
server_key_version: "encryptionkey@0"
I20260928 05:11:33.025990 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:11:33.029457 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:33.030879 8070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:33.031054 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "372ebd49ce994682a3e179a6f9c90f3d"
format_stamp: "Formatted at 2026-09-28 05:11:33 on dist-test-slave-6718"
server_key: "c0dd801e0000795f607cf4d20dd9eac8"
server_key_iv: "714d05f398730ea9b7e6784a0a525549"
server_key_version: "encryptionkey@0"
I20260928 05:11:33.031329 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:33.055797 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:33.059969 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:33.060230 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:33.062834 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:33.062935 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:33.063073 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:33.063145 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:11:33 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:33.087074 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.063351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58049 (local address 127.0.97.62:34727)
0928 05:11:33.064031 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:33.064038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:33.065114 (+ 1076us) server_negotiation.cc:407] Connection header received
0928 05:11:33.068116 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:33.068123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:33.068230 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:33.068352 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:33.070571 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.072048 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.073699 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.074451 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.074618 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.079148 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:33.079157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:33.079162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:33.079209 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:33.082987 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.083915 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.083922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.083927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.084080 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.084580 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.084588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.084592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.084982 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:33.085130 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:33.086172 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 05:11:33.086468 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":443,"threads_started":1}
I20260928 05:11:33.087615 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.064090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.97.1:58049)
0928 05:11:33.064930 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:33.064988 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:33.067758 (+ 2770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:33.068623 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:33.068639 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:33.069439 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:33.070274 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.070307 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.072235 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.072239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.073564 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.073575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.074764 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.074770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.074943 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.075710 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:33.075730 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:33.078904 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:33.083132 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.083144 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.083168 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.083781 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.084214 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.084223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.084227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.084453 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.085125 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:33.085137 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:33.085639 (+ 502us) client_negotiation.cc:769] Sending connection context
0928 05:11:33.085987 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 05:11:33.086417 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260928 05:11:33.108722 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.093241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44913 (local address 127.0.97.62:34727)
0928 05:11:33.093422 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:33.093428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:33.094191 (+ 763us) server_negotiation.cc:407] Connection header received
0928 05:11:33.094330 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:33.094338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:33.094458 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:33.094642 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:33.095560 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.097052 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.099068 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.099568 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.099642 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.102402 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:33.102411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:33.102417 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:33.102465 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:33.105488 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.106373 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.106379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.106382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.106506 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.106963 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.106971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.106977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.107421 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:33.107584 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:33.108290 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 05:11:33.108489 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:11:33.109009 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.093301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.97.1:44913)
0928 05:11:33.094098 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:33.094117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:33.094233 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:33.094629 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:33.094632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:33.095077 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:33.095431 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.095440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.097227 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.097231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.098883 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.098897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.100057 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.100072 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.100230 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.101050 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:33.101079 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:33.102200 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:33.105622 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.105629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.105640 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.106200 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.106632 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.106638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.106643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.106829 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.107666 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:33.107676 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:33.107933 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 05:11:33.108164 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 05:11:33.108500 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":475,"threads_started":1}
I20260928 05:11:33.116421 8070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44877
I20260928 05:11:33.116441 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44877 every 8 connection(s)
I20260928 05:11:33.117571 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:33.119462 8206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34727
I20260928 05:11:33.119633 8206 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:33.120153 8206 heartbeater.cc:511] Master 127.0.97.62:34727 requested a full tablet report, sending...
I20260928 05:11:33.121925 7999 ts_manager.cc:194] Registered new tserver with Master: 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877)
I20260928 05:11:33.123456 7999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58049
I20260928 05:11:33.126940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 8070
I20260928 05:11:33.127130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:33.127756 388 external_mini_cluster.cc:1428] Setting key eaf7aa342a2a53754a56def827f3c0e2
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 05:11:33.172263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34727
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:33.487237 8210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:33.487483 8210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:33.487632 8210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:33.494649 8210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:33.494760 8210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:33.494901 8210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:11:33.495013 8210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:33.509351 8210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:33.510876 8210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:33.512472 8210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:33.526703 8215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:33.526680 8218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:33.526633 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:33.527518 8210 server_base.cc:1034] running on GCE node
I20260928 05:11:33.528074 8210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:33.528923 8210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:33.530117 8210 hybrid_clock.cc:648] HybridClock initialized: now 1790572293530049 us; error 53 us; skew 500 ppm
I20260928 05:11:33.530386 8210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:33 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:33.533370 8210 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:11:33.534822 8210 webserver.cc:460] Webserver started at http://127.0.97.2:35997/ using document root <none> and password file <none>
I20260928 05:11:33.535549 8210 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:33.535650 8210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:33.535950 8210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:33.538700 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "ebe3519d408241ea95f1c840a2aa8895"
format_stamp: "Formatted at 2026-09-28 05:11:33 on dist-test-slave-6718"
server_key: "3e76e4bf214680052d1b9be39fe7b09c"
server_key_iv: "95d85685693dc818b84f83ac48dc8feb"
server_key_version: "encryptionkey@0"
I20260928 05:11:33.539456 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ebe3519d408241ea95f1c840a2aa8895"
format_stamp: "Formatted at 2026-09-28 05:11:33 on dist-test-slave-6718"
server_key: "3e76e4bf214680052d1b9be39fe7b09c"
server_key_iv: "95d85685693dc818b84f83ac48dc8feb"
server_key_version: "encryptionkey@0"
I20260928 05:11:33.544188 8210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 05:11:33.547436 8225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:33.548626 8210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:33.548779 8210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "ebe3519d408241ea95f1c840a2aa8895"
format_stamp: "Formatted at 2026-09-28 05:11:33 on dist-test-slave-6718"
server_key: "3e76e4bf214680052d1b9be39fe7b09c"
server_key_iv: "95d85685693dc818b84f83ac48dc8feb"
server_key_version: "encryptionkey@0"
I20260928 05:11:33.548987 8210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:33.559727 8210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:33.563692 8210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:33.563961 8210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:33.566469 8210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:33.566587 8210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 05:11:33.566736 8210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:33.566823 8210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:11:33 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:33.590651 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.567103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50787 (local address 127.0.97.62:34727)
0928 05:11:33.567264 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:33.567268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:33.568609 (+ 1341us) server_negotiation.cc:407] Connection header received
0928 05:11:33.571574 (+ 2965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:33.571581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:33.571668 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:33.571773 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:33.574048 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.575449 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.577728 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.578290 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.578427 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.583429 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:33.583434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:33.583437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:33.583467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:33.586527 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.587510 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.587517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.587523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.587675 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.588156 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.588165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.588169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.588580 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:33.588756 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:33.589785 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0928 05:11:33.590116 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:11:33.591684 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.567540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.97.2:50787)
0928 05:11:33.568422 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:33.568473 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:33.571222 (+ 2749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:33.572249 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:33.572264 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:33.573006 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:33.573820 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.573872 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.575629 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.575636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.577496 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.577507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.578576 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.578587 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.578929 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.579778 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:33.579814 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:33.583210 (+ 3396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:33.586680 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.586688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.586707 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.587325 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.587812 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.587815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.587817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.587995 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.588749 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:33.588762 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:33.589260 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 05:11:33.589669 (+ 409us) client_negotiation.cc:240] Negotiation successful
0928 05:11:33.590123 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":297,"threads_started":1}
I20260928 05:11:33.612545 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.596852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58821 (local address 127.0.97.62:34727)
0928 05:11:33.597038 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:33.597046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:33.597693 (+ 647us) server_negotiation.cc:407] Connection header received
0928 05:11:33.597869 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:33.597872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:33.597926 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:33.598000 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:33.599412 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.600111 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.602739 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.603495 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.603606 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.606342 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:33.606351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:33.606357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:33.606395 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:33.609883 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.610628 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.610633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.610637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.610761 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:33.611095 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:33.611099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:33.611103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:33.611433 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:33.611568 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:33.611997 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 05:11:33.612200 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:11:33.612648 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:33.596850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.97.2:58821)
0928 05:11:33.597574 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:33.597598 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:33.597757 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:33.598065 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:33.598072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:33.598793 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:33.599256 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.599273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.600239 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.600247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.602518 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:33.602561 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:33.603946 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:33.603955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:33.604190 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:33.605000 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:33.605034 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:33.606151 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:33.609998 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.610003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.610006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.610473 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.610865 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:33.610870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:33.610874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:33.610991 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:33.611614 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:33.611617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:33.611764 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 05:11:33.611903 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 05:11:33.612200 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":358,"threads_started":1}
I20260928 05:11:33.618692 8210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40243
I20260928 05:11:33.618716 8344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40243 every 8 connection(s)
I20260928 05:11:33.619880 8210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:33.621861 8345 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34727
I20260928 05:11:33.621949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 8210
I20260928 05:11:33.622082 8345 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:33.622148 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:33.622632 388 external_mini_cluster.cc:1428] Setting key 145cce950b6caa2f0731b1c9b5cd9ab6
I20260928 05:11:33.622871 8345 heartbeater.cc:511] Master 127.0.97.62:34727 requested a full tablet report, sending...
I20260928 05:11:33.624269 7998 ts_manager.cc:194] Registered new tserver with Master: ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243)
I20260928 05:11:33.625113 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50787
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 05:11:33.666009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34727
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:33.981884 8349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:33.982195 8349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:33.982362 8349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:33.989347 8349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:33.989476 8349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:33.989609 8349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 05:11:33.989735 8349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:33.999904 8349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:34.001201 8349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:34.002568 8349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:34.014786 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:34.015203 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:34.015399 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:34.015837 8349 server_base.cc:1034] running on GCE node
I20260928 05:11:34.016426 8349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:34.017323 8349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:34.018602 8349 hybrid_clock.cc:648] HybridClock initialized: now 1790572294018497 us; error 37 us; skew 500 ppm
I20260928 05:11:34.018899 8349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:34.021914 8349 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:11:34.023389 8349 webserver.cc:460] Webserver started at http://127.0.97.3:33373/ using document root <none> and password file <none>
I20260928 05:11:34.024035 8349 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:34.024154 8349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:34.024459 8349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:34.027314 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "9f02b3a623764b969aa5362ca820004f"
format_stamp: "Formatted at 2026-09-28 05:11:34 on dist-test-slave-6718"
server_key: "2fc884cdb397fdc9f8725156e02d838a"
server_key_iv: "ff8d6b83a4ebf35402642590235dc01c"
server_key_version: "encryptionkey@0"
I20260928 05:11:34.028059 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9f02b3a623764b969aa5362ca820004f"
format_stamp: "Formatted at 2026-09-28 05:11:34 on dist-test-slave-6718"
server_key: "2fc884cdb397fdc9f8725156e02d838a"
server_key_iv: "ff8d6b83a4ebf35402642590235dc01c"
server_key_version: "encryptionkey@0"
I20260928 05:11:34.032825 8349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 05:11:34.036351 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:34.037631 8349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:11:34.037787 8349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "9f02b3a623764b969aa5362ca820004f"
format_stamp: "Formatted at 2026-09-28 05:11:34 on dist-test-slave-6718"
server_key: "2fc884cdb397fdc9f8725156e02d838a"
server_key_iv: "ff8d6b83a4ebf35402642590235dc01c"
server_key_version: "encryptionkey@0"
I20260928 05:11:34.037997 8349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:34.065037 8349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:34.068984 8349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:34.069248 8349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:34.071843 8349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:34.071950 8349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:34.072125 8349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:34.072206 8349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572294, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:34.095208 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.072350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60767 (local address 127.0.97.62:34727)
0928 05:11:34.072534 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.072542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.073789 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 05:11:34.076008 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.076014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.076106 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.076212 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:34.078203 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.079244 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.081689 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.082406 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.082582 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.087670 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:34.087677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:34.087680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:34.087712 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:34.091843 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.092795 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.092800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.092802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.092925 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.093345 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.093351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.093354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.093716 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:34.093855 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.094621 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.094832 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:11:34.096222 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.072879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.97.3:60767)
0928 05:11:34.073637 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.073680 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.075714 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.076485 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.076500 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.077222 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:34.078030 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.078048 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.079415 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.079420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.081516 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.081538 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.082908 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.082919 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.083268 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.084143 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:34.084186 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:34.087392 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:34.092010 (+ 4618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:34.092017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:34.092028 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:34.092597 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:34.093038 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:34.093044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:34.093047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:34.093236 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:34.093846 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:34.093854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:34.094181 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.094555 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.095071 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":317,"threads_started":1}
I20260928 05:11:34.123467 8349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35979
I20260928 05:11:34.123550 8483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35979 every 8 connection(s)
I20260928 05:11:34.125013 8349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:34.125725 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.106848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.97.3:59353)
0928 05:11:34.107429 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.107450 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.107655 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.108243 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.108247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.108983 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:34.109462 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.109481 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.111114 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.111119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.113779 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.113801 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.115146 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.115153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.115360 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.116200 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:34.116225 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:34.117950 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:34.122335 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:34.122348 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:34.122353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:34.122995 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:34.123530 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:34.123541 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:34.123545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:34.123784 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:34.124425 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:34.124433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:34.124665 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.124858 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.125278 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":391,"threads_started":1}
I20260928 05:11:34.125859 8206 heartbeater.cc:503] Master 127.0.97.62:34727 was elected leader, sending a full tablet report...
I20260928 05:11:34.125622 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.106821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59353 (local address 127.0.97.62:34727)
0928 05:11:34.107221 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.107230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.107532 (+ 302us) server_negotiation.cc:407] Connection header received
0928 05:11:34.107840 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.107848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.107934 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.108009 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:34.109664 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.110923 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.113973 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.114708 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.114826 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.118194 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:34.118203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:34.118210 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:34.118254 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:34.122103 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.123156 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.123164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.123169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.123360 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.123920 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.123925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.123929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.124199 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:34.124353 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.125029 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.125278 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260928 05:11:34.129144 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 8349
I20260928 05:11:34.129376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:34.130012 388 external_mini_cluster.cc:1428] Setting key 05e2aee799bdd7e3d2587b7cca07a9a0
I20260928 05:11:34.131101 8484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34727
I20260928 05:11:34.131322 8484 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:34.132022 8484 heartbeater.cc:511] Master 127.0.97.62:34727 requested a full tablet report, sending...
I20260928 05:11:34.133394 7999 ts_manager.cc:194] Registered new tserver with Master: 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:34.134227 7999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60767
I20260928 05:11:34.145147 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:11:34.192276 8094 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:34.182492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.97.62:34727)
0928 05:11:34.182738 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.182746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.182945 (+ 199us) server_negotiation.cc:407] Connection header received
0928 05:11:34.183094 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.183098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.183164 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.183249 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:34.184437 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.185115 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.186316 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.186784 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.186870 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.192016 (+ 5146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37530: BlockingRecv error: recv got EOF from 127.0.0.1:37530 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:34.207512 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.195220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.0.97.62:34727)
0928 05:11:34.195446 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.195452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.195599 (+ 147us) server_negotiation.cc:407] Connection header received
0928 05:11:34.195750 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.195753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.195812 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.195880 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:34.196930 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.197590 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.198880 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.199315 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.199399 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.202803 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:34.202808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:34.202811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:34.202840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:34.205285 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.205880 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.205884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.205886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.205994 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.206307 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.206310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.206312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.206610 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:34.206812 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.207040 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.207183 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 05:11:34.211311 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:34.218109 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:34.246410 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.229414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44877 (local address 127.0.0.1:35978)
0928 05:11:34.236523 (+ 7109us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.236573 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.236781 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.238272 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.238279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.238306 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.238727 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.238740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.240784 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.240790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.243408 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.243421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.245363 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.245370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.245564 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.245621 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.245738 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.245940 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6758,"mutex_wait_us":84,"thread_start_us":8220,"threads_started":1}
I20260928 05:11:34.247243 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.230082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.0.97.3:35979)
0928 05:11:34.237467 (+ 7385us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.237482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.238117 (+ 635us) server_negotiation.cc:407] Connection header received
0928 05:11:34.238230 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.238238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.238472 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.238749 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.239707 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.241369 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.243871 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.245216 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.245350 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.245589 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.246639 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.246807 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7151,"thread_start_us":6897,"threads_started":1}
I20260928 05:11:34.246498 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.229558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35979 (local address 127.0.0.1:58658)
0928 05:11:34.237877 (+ 8319us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.237893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.237999 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.239040 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.239044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.239067 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.239482 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.239495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.241529 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.241534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.243709 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.243720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.245732 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.245737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.245923 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.245951 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.246030 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.246155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8212,"thread_start_us":6304,"threads_started":1}
I20260928 05:11:34.247649 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.229637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.0.97.1:44877)
0928 05:11:34.237625 (+ 7988us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.237635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.237668 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:11:34.237769 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.237777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.238045 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.238924 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.239103 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.240637 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.243600 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.245140 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.246759 (+ 1619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.247099 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.247189 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.247361 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7825,"thread_start_us":7707,"threads_started":1}
I20260928 05:11:34.251631 8136 tablet_service.cc:1467] Processing CreateTablet for tablet d0c7ff1166fa4cbf8429514d1c958fba (DEFAULT_TABLE table=test-table [id=3f043fa06c0646c68c832f61da8d463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:34.251639 8414 tablet_service.cc:1467] Processing CreateTablet for tablet d0c7ff1166fa4cbf8429514d1c958fba (DEFAULT_TABLE table=test-table [id=3f043fa06c0646c68c832f61da8d463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:34.253288 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c7ff1166fa4cbf8429514d1c958fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.253288 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c7ff1166fa4cbf8429514d1c958fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.254747 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.238334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.0.97.2:40243)
0928 05:11:34.248115 (+ 9781us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.248123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.248148 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:11:34.248291 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.248298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.248535 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.248738 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.249318 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.250672 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.252286 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.253527 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.253645 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.253862 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.254293 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.254448 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9618,"thread_start_us":10138,"threads_started":1}
I20260928 05:11:34.254746 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.238035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40243 (local address 127.0.0.1:49076)
0928 05:11:34.244819 (+ 6784us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.244846 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.245000 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.248903 (+ 3903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.248906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.248926 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.249170 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.249177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.250797 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.250802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.252129 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.252140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.253990 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.253996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.254122 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.254145 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.254251 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.254378 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6602,"thread_start_us":7072,"threads_started":1}
I20260928 05:11:34.258098 8275 tablet_service.cc:1467] Processing CreateTablet for tablet d0c7ff1166fa4cbf8429514d1c958fba (DEFAULT_TABLE table=test-table [id=3f043fa06c0646c68c832f61da8d463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:34.259300 8275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c7ff1166fa4cbf8429514d1c958fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.270098 8508 tablet_bootstrap.cc:492] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Bootstrap starting.
I20260928 05:11:34.271108 8509 tablet_bootstrap.cc:492] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Bootstrap starting.
I20260928 05:11:34.272176 8510 tablet_bootstrap.cc:492] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Bootstrap starting.
I20260928 05:11:34.274395 8508 tablet_bootstrap.cc:654] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.275305 8510 tablet_bootstrap.cc:654] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.275507 8508 log.cc:824] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:34.276726 8510 log.cc:824] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:34.278329 8509 tablet_bootstrap.cc:654] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.278847 8508 tablet_bootstrap.cc:492] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: No bootstrap required, opened a new log
I20260928 05:11:34.279132 8508 ts_tablet_manager.cc:1397] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260928 05:11:34.279629 8509 log.cc:824] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:34.280999 8510 tablet_bootstrap.cc:492] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: No bootstrap required, opened a new log
I20260928 05:11:34.281253 8510 ts_tablet_manager.cc:1397] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 05:11:34.282286 8509 tablet_bootstrap.cc:492] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: No bootstrap required, opened a new log
I20260928 05:11:34.282588 8509 ts_tablet_manager.cc:1397] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260928 05:11:34.289647 8508 raft_consensus.cc:348] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.290109 8508 raft_consensus.cc:374] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.290287 8508 raft_consensus.cc:729] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.290518 8509 raft_consensus.cc:348] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.291049 8509 raft_consensus.cc:374] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.290940 8508 consensus_queue.cc:260] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.291172 8509 raft_consensus.cc:729] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.291525 8510 raft_consensus.cc:348] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.291942 8510 raft_consensus.cc:374] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.292037 8510 raft_consensus.cc:729] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3519d408241ea95f1c840a2aa8895, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.292721 8508 ts_tablet_manager.cc:1428] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260928 05:11:34.292649 8510 consensus_queue.cc:260] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.293455 8509 consensus_queue.cc:260] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.293723 8345 heartbeater.cc:503] Master 127.0.97.62:34727 was elected leader, sending a full tablet report...
I20260928 05:11:34.294435 8510 ts_tablet_manager.cc:1428] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Time spent starting tablet: real 0.013s user 0.006s sys 0.009s
I20260928 05:11:34.296159 8484 heartbeater.cc:503] Master 127.0.97.62:34727 was elected leader, sending a full tablet report...
I20260928 05:11:34.298032 8509 ts_tablet_manager.cc:1428] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
W20260928 05:11:34.370699 8207 tablet.cc:2369] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:11:34.371747 8346 tablet.cc:2369] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:11:34.383231 8485 tablet.cc:2369] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:11:34.442471 8516 raft_consensus.cc:482] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:34.442827 8516 raft_consensus.cc:504] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.444453 8516 leader_election.cc:288] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877), ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243)
I20260928 05:11:34.450372 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.444789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44877 (local address 127.0.97.3:48333)
0928 05:11:34.444986 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.445006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.445136 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.445503 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.445507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.445535 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.445798 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.445812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.447050 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.447053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.448706 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.448722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.449857 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.449861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.449998 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.450021 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.450091 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.450177 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 05:11:34.450500 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.444885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48333 (local address 127.0.97.1:44877)
0928 05:11:34.445054 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.445058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.445075 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:11:34.445280 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.445286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.445375 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.445479 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.445986 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.446933 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.448869 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.449718 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.449846 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.449949 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.450133 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.450259 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:11:34.451493 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.445359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39509 (local address 127.0.97.2:40243)
0928 05:11:34.445598 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.445607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.446171 (+ 564us) server_negotiation.cc:407] Connection header received
0928 05:11:34.446260 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.446266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.446338 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.446431 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.446935 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.447647 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.449543 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.450716 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.450850 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.450946 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.451236 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.451324 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:11:34.451648 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.445235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40243 (local address 127.0.97.3:39509)
0928 05:11:34.446076 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.446090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.446177 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.446447 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.446449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.446465 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.446786 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.446805 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.447775 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.447781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.449398 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.449408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.450904 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.450908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.451054 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.451092 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.451192 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.451323 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":507,"threads_started":1}
I20260928 05:11:34.451507 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c7ff1166fa4cbf8429514d1c958fba" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ebd49ce994682a3e179a6f9c90f3d" is_pre_election: true
I20260928 05:11:34.452037 8161 raft_consensus.cc:2454] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 0.
I20260928 05:11:34.452632 8366 leader_election.cc:302] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, 9f02b3a623764b969aa5362ca820004f; no voters:
I20260928 05:11:34.452553 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c7ff1166fa4cbf8429514d1c958fba" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895" is_pre_election: true
I20260928 05:11:34.453006 8516 raft_consensus.cc:2781] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:34.453011 8298 raft_consensus.cc:2454] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 0.
I20260928 05:11:34.453187 8516 raft_consensus.cc:482] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:34.453313 8516 raft_consensus.cc:3037] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.455497 8516 raft_consensus.cc:504] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.456063 8516 leader_election.cc:288] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 election: Requested vote from peers 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877), ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243)
I20260928 05:11:34.456419 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c7ff1166fa4cbf8429514d1c958fba" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ebd49ce994682a3e179a6f9c90f3d"
I20260928 05:11:34.456482 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c7ff1166fa4cbf8429514d1c958fba" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895"
I20260928 05:11:34.456627 8161 raft_consensus.cc:3037] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.456687 8298 raft_consensus.cc:3037] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.459802 8298 raft_consensus.cc:2454] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 1.
I20260928 05:11:34.459802 8161 raft_consensus.cc:2454] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 1.
I20260928 05:11:34.460160 8366 leader_election.cc:302] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f02b3a623764b969aa5362ca820004f, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:34.460426 8516 raft_consensus.cc:2781] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:34.460824 8516 raft_consensus.cc:686] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 1 LEADER]: Becoming Leader. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Running, Role: LEADER
I20260928 05:11:34.461264 8516 consensus_queue.cc:237] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } }
I20260928 05:11:34.465723 7999 catalog_manager.cc:5495] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f02b3a623764b969aa5362ca820004f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9f02b3a623764b969aa5362ca820004f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } health_report { overall_health: UNKNOWN } } }
I20260928 05:11:34.493764 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.488454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.97.3:35979)
0928 05:11:34.488641 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.488647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.488666 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:11:34.488883 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.488887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.488958 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.489104 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:34.489459 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.490205 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.491686 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.492081 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.492214 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.492654 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:34.493238 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:34.493391 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.493447 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.493523 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:11:34.502009 8161 raft_consensus.cc:1261] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Refusing update from remote peer 9f02b3a623764b969aa5362ca820004f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:34.502009 8298 raft_consensus.cc:1261] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Refusing update from remote peer 9f02b3a623764b969aa5362ca820004f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:34.502974 8516 consensus_queue.cc:1035] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:34.503300 8518 consensus_queue.cc:1035] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [LEADER]: Connected to new peer: Peer: permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:34.515071 8521 mvcc.cc:205] Tried to move back new op lower bound from 7334184118269898752 to 7334184118116790272. Current Snapshot: MvccSnapshot[applied={T|T < 7334184118269898752}]
I20260928 05:11:34.531168 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:34.531993 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:34.535604 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Sending DeleteTablet for 3 replicas of tablet d0c7ff1166fa4cbf8429514d1c958fba
I20260928 05:11:34.536573 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c7ff1166fa4cbf8429514d1c958fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:34 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35978
I20260928 05:11:34.536690 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c7ff1166fa4cbf8429514d1c958fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:34 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58658
I20260928 05:11:34.536963 8275 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c7ff1166fa4cbf8429514d1c958fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:34 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49076
I20260928 05:11:34.541297 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:34.542567 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:34.551436 8135 tablet_service.cc:1467] Processing CreateTablet for tablet fc9f8824a3e34eb98c451a49bc3a28d1 (DEFAULT_TABLE table=test-table [id=a46121be480e4a4c92b67e6642bf4db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:34.551995 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9f8824a3e34eb98c451a49bc3a28d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.551996 8413 tablet_service.cc:1467] Processing CreateTablet for tablet fc9f8824a3e34eb98c451a49bc3a28d1 (DEFAULT_TABLE table=test-table [id=a46121be480e4a4c92b67e6642bf4db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:34.552588 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9f8824a3e34eb98c451a49bc3a28d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.554765 8531 tablet_replica.cc:330] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: stopping tablet replica
I20260928 05:11:34.555279 8531 raft_consensus.cc:2229] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:34.556383 8274 tablet_service.cc:1467] Processing CreateTablet for tablet fc9f8824a3e34eb98c451a49bc3a28d1 (DEFAULT_TABLE table=test-table [id=a46121be480e4a4c92b67e6642bf4db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:34.557165 8530 tablet_replica.cc:330] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: stopping tablet replica
I20260928 05:11:34.557617 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9f8824a3e34eb98c451a49bc3a28d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.557628 8530 raft_consensus.cc:2229] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:34.557739 8532 tablet_replica.cc:330] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: stopping tablet replica
I20260928 05:11:34.558187 8532 raft_consensus.cc:2229] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:34.558734 8532 raft_consensus.cc:2258] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:34.560627 8508 tablet_bootstrap.cc:492] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Bootstrap starting.
I20260928 05:11:34.560900 8532 ts_tablet_manager.cc:1905] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:34.562134 8508 tablet_bootstrap.cc:654] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.562491 8509 tablet_bootstrap.cc:492] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Bootstrap starting.
I20260928 05:11:34.563982 8509 tablet_bootstrap.cc:654] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.566346 8510 tablet_bootstrap.cc:492] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Bootstrap starting.
I20260928 05:11:34.568600 8510 tablet_bootstrap.cc:654] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.562134 8531 raft_consensus.cc:2258] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:34.569011 8532 ts_tablet_manager.cc:1918] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:34.569166 8532 log.cc:1196] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/d0c7ff1166fa4cbf8429514d1c958fba
I20260928 05:11:34.569664 8532 ts_tablet_manager.cc:1939] T d0c7ff1166fa4cbf8429514d1c958fba P ebe3519d408241ea95f1c840a2aa8895: Deleting consensus metadata
I20260928 05:11:34.558266 8530 raft_consensus.cc:2258] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:34.571352 8509 tablet_bootstrap.cc:492] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: No bootstrap required, opened a new log
I20260928 05:11:34.571529 8509 ts_tablet_manager.cc:1397] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 05:11:34.571585 7984 catalog_manager.cc:4841] TS ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243): tablet d0c7ff1166fa4cbf8429514d1c958fba (table test-table [id=3f043fa06c0646c68c832f61da8d463f]) successfully deleted
I20260928 05:11:34.572489 8509 raft_consensus.cc:348] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.572744 8509 raft_consensus.cc:374] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.572847 8509 raft_consensus.cc:729] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.573127 8509 consensus_queue.cc:260] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.573416 8530 ts_tablet_manager.cc:1905] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:34.573787 8509 ts_tablet_manager.cc:1428] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:11:34.580245 8530 ts_tablet_manager.cc:1918] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:34.580399 8530 log.cc:1196] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/d0c7ff1166fa4cbf8429514d1c958fba
I20260928 05:11:34.580916 8530 ts_tablet_manager.cc:1939] T d0c7ff1166fa4cbf8429514d1c958fba P 372ebd49ce994682a3e179a6f9c90f3d: Deleting consensus metadata
I20260928 05:11:34.582552 7984 catalog_manager.cc:4841] TS 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877): tablet d0c7ff1166fa4cbf8429514d1c958fba (table test-table [id=3f043fa06c0646c68c832f61da8d463f]) successfully deleted
I20260928 05:11:34.585497 8531 ts_tablet_manager.cc:1905] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:34.590909 8510 tablet_bootstrap.cc:492] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: No bootstrap required, opened a new log
I20260928 05:11:34.591104 8510 ts_tablet_manager.cc:1397] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20260928 05:11:34.591949 8510 raft_consensus.cc:348] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.592248 8510 raft_consensus.cc:374] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.592384 8510 raft_consensus.cc:729] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3519d408241ea95f1c840a2aa8895, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.592721 8510 consensus_queue.cc:260] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.593444 8510 ts_tablet_manager.cc:1428] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:34.595098 8508 tablet_bootstrap.cc:492] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: No bootstrap required, opened a new log
I20260928 05:11:34.595294 8508 ts_tablet_manager.cc:1397] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.014s
I20260928 05:11:34.596168 8508 raft_consensus.cc:348] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.596447 8508 raft_consensus.cc:374] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.596580 8508 raft_consensus.cc:729] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.596815 8531 ts_tablet_manager.cc:1918] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:34.596998 8531 log.cc:1196] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/d0c7ff1166fa4cbf8429514d1c958fba
I20260928 05:11:34.596889 8508 consensus_queue.cc:260] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.597577 8531 ts_tablet_manager.cc:1939] T d0c7ff1166fa4cbf8429514d1c958fba P 9f02b3a623764b969aa5362ca820004f: Deleting consensus metadata
I20260928 05:11:34.597584 8508 ts_tablet_manager.cc:1428] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:34.599711 7985 catalog_manager.cc:4841] TS 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979): tablet d0c7ff1166fa4cbf8429514d1c958fba (table test-table [id=3f043fa06c0646c68c832f61da8d463f]) successfully deleted
I20260928 05:11:34.837011 8516 raft_consensus.cc:482] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:34.837241 8516 raft_consensus.cc:504] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.837904 8516 leader_election.cc:288] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877), ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243)
I20260928 05:11:34.838605 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9f8824a3e34eb98c451a49bc3a28d1" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895" is_pre_election: true
I20260928 05:11:34.838587 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9f8824a3e34eb98c451a49bc3a28d1" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ebd49ce994682a3e179a6f9c90f3d" is_pre_election: true
I20260928 05:11:34.838857 8298 raft_consensus.cc:2454] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 0.
I20260928 05:11:34.838852 8161 raft_consensus.cc:2454] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 0.
I20260928 05:11:34.839285 8366 leader_election.cc:302] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f02b3a623764b969aa5362ca820004f, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:34.839562 8516 raft_consensus.cc:2781] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:34.839700 8516 raft_consensus.cc:482] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:34.839826 8516 raft_consensus.cc:3037] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.842260 8516 raft_consensus.cc:504] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.842806 8516 leader_election.cc:288] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 election: Requested vote from peers 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877), ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243)
I20260928 05:11:34.843348 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9f8824a3e34eb98c451a49bc3a28d1" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895"
I20260928 05:11:34.843400 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9f8824a3e34eb98c451a49bc3a28d1" candidate_uuid: "9f02b3a623764b969aa5362ca820004f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ebd49ce994682a3e179a6f9c90f3d"
I20260928 05:11:34.843523 8298 raft_consensus.cc:3037] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.843593 8161 raft_consensus.cc:3037] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.845433 8298 raft_consensus.cc:2454] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 1.
I20260928 05:11:34.845836 8366 leader_election.cc:302] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f02b3a623764b969aa5362ca820004f, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:34.846107 8516 raft_consensus.cc:2781] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:34.846263 8516 raft_consensus.cc:686] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 1 LEADER]: Becoming Leader. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Running, Role: LEADER
I20260928 05:11:34.846570 8161 raft_consensus.cc:2454] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f02b3a623764b969aa5362ca820004f in term 1.
I20260928 05:11:34.846588 8516 consensus_queue.cc:237] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.849198 7999 catalog_manager.cc:5495] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f02b3a623764b969aa5362ca820004f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9f02b3a623764b969aa5362ca820004f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } health_report { overall_health: HEALTHY } } }
Sep 28 05:11:34 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:34.906075 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.888765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.0.97.62:34727)
0928 05:11:34.889719 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.889723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.889738 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:11:34.889774 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.889777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.889833 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.889944 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:34.891072 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.891858 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.893139 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.893577 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.893708 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.900873 (+ 7165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:34.900879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:34.900882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:34.900909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:34.903495 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.904147 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.904153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.904159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.904274 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:34.904629 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:34.904636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:34.904639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:34.904901 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:34.905045 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.905434 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.905742 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":824,"thread_start_us":692,"threads_started":1}
I20260928 05:11:34.906083 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.888714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34727 (local address 127.0.0.1:37546)
0928 05:11:34.889373 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.889400 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.889557 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.889964 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.889969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.890757 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:34.890943 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.890952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.891991 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.891998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.893016 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.893027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.893724 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.893729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.893908 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.898560 (+ 4652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:34.898593 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:34.900672 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:34.903631 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:34.903643 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:34.903651 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:34.904035 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:34.904401 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:34.904408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:34.904411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:34.904532 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:34.905022 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:34.905030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:34.905184 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.905347 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.905627 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":356,"threads_started":1}
I20260928 05:11:34.911147 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37546:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 05:11:34.912137 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:34.919051 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 8234dc7855704f1e89661e9386977c44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69c0fffecad74bc685948283830b7cf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:11:34.919070 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 8234dc7855704f1e89661e9386977c44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69c0fffecad74bc685948283830b7cf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:11:34.919575 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8234dc7855704f1e89661e9386977c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.919580 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8234dc7855704f1e89661e9386977c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.924223 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 8234dc7855704f1e89661e9386977c44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69c0fffecad74bc685948283830b7cf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 05:11:34.924809 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8234dc7855704f1e89661e9386977c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:34.926703 8509 tablet_bootstrap.cc:492] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: Bootstrap starting.
I20260928 05:11:34.928056 8509 tablet_bootstrap.cc:654] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.929220 8510 tablet_bootstrap.cc:492] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: Bootstrap starting.
I20260928 05:11:34.930519 8510 tablet_bootstrap.cc:654] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.931318 8509 tablet_bootstrap.cc:492] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: No bootstrap required, opened a new log
I20260928 05:11:34.931571 8509 ts_tablet_manager.cc:1397] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260928 05:11:34.932579 8508 tablet_bootstrap.cc:492] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: Bootstrap starting.
I20260928 05:11:34.932559 8509 raft_consensus.cc:348] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.933036 8509 raft_consensus.cc:374] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.933195 8509 raft_consensus.cc:729] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.933573 8509 consensus_queue.cc:260] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.934259 8516 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:34.934715 8516 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:34.935120 8508 tablet_bootstrap.cc:654] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:34.935398 8509 ts_tablet_manager.cc:1428] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260928 05:11:34.938514 8508 tablet_bootstrap.cc:492] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: No bootstrap required, opened a new log
I20260928 05:11:34.938701 8508 ts_tablet_manager.cc:1397] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 05:11:34.939798 8508 raft_consensus.cc:348] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.940068 8508 raft_consensus.cc:374] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.940140 8508 raft_consensus.cc:729] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.940356 8508 consensus_queue.cc:260] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.941054 8508 ts_tablet_manager.cc:1428] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:11:34.940936 8514 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:34.941407 8514 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:34.942569 8510 tablet_bootstrap.cc:492] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: No bootstrap required, opened a new log
I20260928 05:11:34.942735 8510 ts_tablet_manager.cc:1397] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260928 05:11:34.943390 8510 raft_consensus.cc:348] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.943617 8510 raft_consensus.cc:374] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:34.943714 8510 raft_consensus.cc:729] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3519d408241ea95f1c840a2aa8895, State: Initialized, Role: FOLLOWER
I20260928 05:11:34.943959 8510 consensus_queue.cc:260] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.944568 8510 ts_tablet_manager.cc:1428] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 05:11:34.944483 8515 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:34.944861 8515 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:34.957693 8514 raft_consensus.cc:482] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:34.957927 8514 raft_consensus.cc:504] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.959476 8514 leader_election.cc:288] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243), 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:34.977384 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.959833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45089 (local address 127.0.97.2:40243)
0928 05:11:34.971317 (+ 11484us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.971324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.971345 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:11:34.971621 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.971627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.971722 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.971780 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.972515 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.973604 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.975587 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.976341 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.976485 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.976598 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.976936 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.977070 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11362,"thread_start_us":11605,"threads_started":1}
I20260928 05:11:34.977384 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.959600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40243 (local address 127.0.97.1:45089)
0928 05:11:34.971078 (+ 11478us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.971106 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.971311 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.972020 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.972024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.972051 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.972341 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.972356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.973777 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.973782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.975423 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.975436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.976522 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.976530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.976744 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.976783 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.976901 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.977038 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11305,"thread_start_us":11887,"threads_started":1}
I20260928 05:11:34.978107 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8234dc7855704f1e89661e9386977c44" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895" is_pre_election: true
I20260928 05:11:34.978404 8298 raft_consensus.cc:2454] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 0.
I20260928 05:11:34.979060 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.959733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35979 (local address 127.0.97.1:41651)
0928 05:11:34.971770 (+ 12037us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:34.971788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:34.971918 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:34.972283 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:34.972290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:34.972320 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:34.972732 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.972743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.974133 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.974138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.976407 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:34.976421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.978154 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.978160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:34.978348 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.978386 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 05:11:34.978497 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 05:11:34.978656 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11920,"mutex_wait_us":1,"thread_start_us":11047,"threads_started":1}
I20260928 05:11:34.979103 8087 leader_election.cc:302] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:34.979329 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:34.960128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41651 (local address 127.0.97.3:35979)
0928 05:11:34.960379 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:34.960388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:34.971901 (+ 11513us) server_negotiation.cc:407] Connection header received
0928 05:11:34.972048 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:34.972053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:34.972131 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:34.972241 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:34.972922 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.973993 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.976630 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:34.977905 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:34.978028 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:34.978119 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:34.978612 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 05:11:34.978770 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:11:34.979542 8514 raft_consensus.cc:2781] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:34.979688 8514 raft_consensus.cc:482] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:34.979813 8514 raft_consensus.cc:3037] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.980136 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8234dc7855704f1e89661e9386977c44" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02b3a623764b969aa5362ca820004f" is_pre_election: true
I20260928 05:11:34.980527 8438 raft_consensus.cc:2454] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 0.
I20260928 05:11:34.982661 8514 raft_consensus.cc:504] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.983280 8514 leader_election.cc:288] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 election: Requested vote from peers ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243), 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:34.983737 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8234dc7855704f1e89661e9386977c44" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895"
I20260928 05:11:34.983759 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8234dc7855704f1e89661e9386977c44" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02b3a623764b969aa5362ca820004f"
I20260928 05:11:34.983966 8438 raft_consensus.cc:3037] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.983980 8298 raft_consensus.cc:3037] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:34.986964 8298 raft_consensus.cc:2454] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 1.
I20260928 05:11:34.987020 8438 raft_consensus.cc:2454] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 1.
I20260928 05:11:34.987459 8088 leader_election.cc:302] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, 9f02b3a623764b969aa5362ca820004f; no voters:
I20260928 05:11:34.987777 8514 raft_consensus.cc:2781] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:34.988220 8514 raft_consensus.cc:686] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 LEADER]: Becoming Leader. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Running, Role: LEADER
I20260928 05:11:34.988639 8514 consensus_queue.cc:237] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:34.990563 8550 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: TxnStatusTablet state changed. Reason: New leader 372ebd49ce994682a3e179a6f9c90f3d. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:34.990953 8550 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:11:34.991658 8552 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 05:11:34.991873 8552 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 05:11:34.992384 7997 catalog_manager.cc:5495] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } health_report { overall_health: UNKNOWN } } }
I20260928 05:11:35.024639 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.017897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44877 (local address 127.0.0.1:35984)
0928 05:11:35.018243 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.018268 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.018409 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.018972 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.018976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.018986 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:11:35.019139 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.019146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.020700 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.020704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.021751 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.021762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.022773 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.022778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.022943 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.023031 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.024139 (+ 1108us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.024155 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.024253 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.024361 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 05:11:35.024809 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.017875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.0.97.1:44877)
0928 05:11:35.018524 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.018562 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.018586 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:35.018677 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.018683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.018778 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.018948 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.019312 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.020564 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.021896 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.022622 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.022752 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.023252 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.024029 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.024173 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.024289 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.024381 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":360,"threads_started":1}
I20260928 05:11:35.029327 8298 raft_consensus.cc:1261] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Refusing update from remote peer 372ebd49ce994682a3e179a6f9c90f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.029592 8438 raft_consensus.cc:1261] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Refusing update from remote peer 372ebd49ce994682a3e179a6f9c90f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.030180 8550 consensus_queue.cc:1035] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.030622 8514 consensus_queue.cc:1035] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.042781 8516 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: TxnStatusTablet state changed. Reason: New leader 372ebd49ce994682a3e179a6f9c90f3d. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:35.043138 8516 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:35.043125 8515 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: TxnStatusTablet state changed. Reason: New leader 372ebd49ce994682a3e179a6f9c90f3d. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:35.043471 8515 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:35.047354 8556 mvcc.cc:205] Tried to move back new op lower bound from 7334184120432402432 to 7334184120277307392. Current Snapshot: MvccSnapshot[applied={T|T < 7334184120432402432}]
I20260928 05:11:35.047744 8515 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:35.047996 8515 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P ebe3519d408241ea95f1c840a2aa8895: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:35.051726 8514 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:35.052134 8514 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:11:35.053471 8514 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:35.053769 8514 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 372ebd49ce994682a3e179a6f9c90f3d: This TxnStatusTablet replica's current role is: LEADER
I20260928 05:11:35.059468 8516 tablet_replica.cc:439] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } } }
I20260928 05:11:35.059788 8516 tablet_replica.cc:442] T 8234dc7855704f1e89661e9386977c44 P 9f02b3a623764b969aa5362ca820004f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 05:11:35.071954 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.065669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44877 (local address 127.0.97.3:56497)
0928 05:11:35.066357 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.066381 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.066519 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.066913 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.066918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.066931 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:11:35.067210 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.067221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.068252 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.068258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.069967 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.069982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.070932 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.070937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.071057 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.071107 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.071500 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.071508 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.071570 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.071675 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":351,"threads_started":1}
I20260928 05:11:35.072003 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.065662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56497 (local address 127.0.97.1:44877)
0928 05:11:35.065868 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.065881 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.066470 (+ 589us) server_negotiation.cc:407] Connection header received
0928 05:11:35.066683 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.066691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.066786 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.066894 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.067348 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.068121 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.070107 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.070696 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.070826 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.071267 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.071399 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.071493 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.071622 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.071699 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:11:35.080561 8298 raft_consensus.cc:1261] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Refusing update from remote peer 9f02b3a623764b969aa5362ca820004f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.080821 8161 raft_consensus.cc:1261] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Refusing update from remote peer 9f02b3a623764b969aa5362ca820004f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.081071 8518 consensus_queue.cc:1035] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.081377 8516 consensus_queue.cc:1035] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [LEADER]: Connected to new peer: Peer: permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.128921 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.120993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39785 (local address 127.0.97.3:35979)
0928 05:11:35.121347 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.121353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.122118 (+ 765us) server_negotiation.cc:407] Connection header received
0928 05:11:35.122292 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.122298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.122382 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.122467 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.123457 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.124547 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.126737 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.127399 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.127491 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.128146 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.128281 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.128346 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.128556 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.128634 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:11:35.129885 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.121378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35979 (local address 127.0.97.1:39785)
0928 05:11:35.121993 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.122013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.122162 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.122647 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.122652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.122666 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:11:35.123059 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.123071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.124723 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.124729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.126565 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.126579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.127697 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.127703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.127896 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.127962 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.128433 (+ 471us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.128443 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.129357 (+ 914us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.129499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":233,"threads_started":1}
I20260928 05:11:35.183475 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:35.183710 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:35.186522 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Sending DeleteTablet for 3 replicas of tablet fc9f8824a3e34eb98c451a49bc3a28d1
I20260928 05:11:35.187457 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9f8824a3e34eb98c451a49bc3a28d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58658
I20260928 05:11:35.188125 8574 tablet_replica.cc:330] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: stopping tablet replica
I20260928 05:11:35.188138 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9f8824a3e34eb98c451a49bc3a28d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49076
I20260928 05:11:35.188476 8574 raft_consensus.cc:2229] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:35.188920 8574 raft_consensus.cc:2258] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.190956 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9f8824a3e34eb98c451a49bc3a28d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35978
I20260928 05:11:35.191148 8574 ts_tablet_manager.cc:1905] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.192361 8576 tablet_replica.cc:330] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: stopping tablet replica
I20260928 05:11:35.192556 8576 raft_consensus.cc:2229] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:35.192874 8576 raft_consensus.cc:2258] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.194060 8576 ts_tablet_manager.cc:1905] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.199988 8574 ts_tablet_manager.cc:1918] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:11:35.200233 8574 log.cc:1196] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/fc9f8824a3e34eb98c451a49bc3a28d1
I20260928 05:11:35.200842 8574 ts_tablet_manager.cc:1939] T fc9f8824a3e34eb98c451a49bc3a28d1 P 9f02b3a623764b969aa5362ca820004f: Deleting consensus metadata
I20260928 05:11:35.201211 8576 ts_tablet_manager.cc:1918] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:11:35.201378 8576 log.cc:1196] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/fc9f8824a3e34eb98c451a49bc3a28d1
I20260928 05:11:35.201825 8576 ts_tablet_manager.cc:1939] T fc9f8824a3e34eb98c451a49bc3a28d1 P 372ebd49ce994682a3e179a6f9c90f3d: Deleting consensus metadata
I20260928 05:11:35.202498 7985 catalog_manager.cc:4841] TS 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979): tablet fc9f8824a3e34eb98c451a49bc3a28d1 (table test-table [id=a46121be480e4a4c92b67e6642bf4db2]) successfully deleted
I20260928 05:11:35.202844 7984 catalog_manager.cc:4841] TS 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877): tablet fc9f8824a3e34eb98c451a49bc3a28d1 (table test-table [id=a46121be480e4a4c92b67e6642bf4db2]) successfully deleted
I20260928 05:11:35.204033 8575 tablet_replica.cc:330] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: stopping tablet replica
I20260928 05:11:35.204275 8575 raft_consensus.cc:2229] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:35.204543 8575 raft_consensus.cc:2258] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.205591 8575 ts_tablet_manager.cc:1905] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.212651 8575 ts_tablet_manager.cc:1918] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:11:35.212793 8575 log.cc:1196] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/fc9f8824a3e34eb98c451a49bc3a28d1
I20260928 05:11:35.213219 8575 ts_tablet_manager.cc:1939] T fc9f8824a3e34eb98c451a49bc3a28d1 P ebe3519d408241ea95f1c840a2aa8895: Deleting consensus metadata
I20260928 05:11:35.214334 7984 catalog_manager.cc:4841] TS ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243): tablet fc9f8824a3e34eb98c451a49bc3a28d1 (table test-table [id=a46121be480e4a4c92b67e6642bf4db2]) successfully deleted
Sep 28 05:11:35 dist-test-slave-6718 krb5kdc[7954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 05:11:35 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:35 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:11:35.252802 8543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:35.241345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.97.62:34727)
0928 05:11:35.241589 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.241596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.242010 (+ 414us) server_negotiation.cc:407] Connection header received
0928 05:11:35.242190 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.242194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.242318 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.242505 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:35.244157 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.244892 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.246735 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.247184 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.247295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.252528 (+ 5233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37558: BlockingRecv error: recv got EOF from 127.0.0.1:37558 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 05:11:35 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 05:11:35.272313 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.255641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.0.97.62:34727)
0928 05:11:35.255900 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.255908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.258443 (+ 2535us) server_negotiation.cc:407] Connection header received
0928 05:11:35.258602 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.258610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.258690 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.258792 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:35.259866 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.260965 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.262166 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.262841 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.262965 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.266856 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:35.266861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:35.266864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:35.266911 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:35.269684 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:35.270447 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:35.270451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:35.270454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:35.270654 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:35.271042 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:35.271045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:35.271046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:35.271317 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:35.271491 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.271870 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.272029 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:11:35.275740 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:11:35.276360 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:35.283257 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 93d14022a6244709b787787e265b66dc (DEFAULT_TABLE table=test-table [id=734ef0c6bf164d9f8edd00b879504309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:35.283263 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 93d14022a6244709b787787e265b66dc (DEFAULT_TABLE table=test-table [id=734ef0c6bf164d9f8edd00b879504309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:35.283702 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d14022a6244709b787787e265b66dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:35.283701 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d14022a6244709b787787e265b66dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:35.288511 8509 tablet_bootstrap.cc:492] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Bootstrap starting.
I20260928 05:11:35.289649 8510 tablet_bootstrap.cc:492] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Bootstrap starting.
I20260928 05:11:35.289958 8509 tablet_bootstrap.cc:654] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:35.289983 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 93d14022a6244709b787787e265b66dc (DEFAULT_TABLE table=test-table [id=734ef0c6bf164d9f8edd00b879504309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:35.290387 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d14022a6244709b787787e265b66dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:35.291303 8510 tablet_bootstrap.cc:654] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:35.297199 8508 tablet_bootstrap.cc:492] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Bootstrap starting.
I20260928 05:11:35.298938 8508 tablet_bootstrap.cc:654] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:35.302651 8510 tablet_bootstrap.cc:492] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: No bootstrap required, opened a new log
I20260928 05:11:35.302651 8509 tablet_bootstrap.cc:492] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: No bootstrap required, opened a new log
I20260928 05:11:35.302801 8509 ts_tablet_manager.cc:1397] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 05:11:35.302944 8510 ts_tablet_manager.cc:1397] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260928 05:11:35.303594 8509 raft_consensus.cc:348] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.303862 8509 raft_consensus.cc:374] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:35.303951 8509 raft_consensus.cc:729] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Initialized, Role: FOLLOWER
I20260928 05:11:35.303691 8510 raft_consensus.cc:348] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.304224 8510 raft_consensus.cc:374] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:35.304355 8510 raft_consensus.cc:729] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3519d408241ea95f1c840a2aa8895, State: Initialized, Role: FOLLOWER
I20260928 05:11:35.304224 8509 consensus_queue.cc:260] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.304680 8510 consensus_queue.cc:260] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.305788 8509 ts_tablet_manager.cc:1428] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 05:11:35.308959 8510 ts_tablet_manager.cc:1428] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 05:11:35.310969 8508 tablet_bootstrap.cc:492] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: No bootstrap required, opened a new log
I20260928 05:11:35.311142 8508 ts_tablet_manager.cc:1397] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260928 05:11:35.311995 8508 raft_consensus.cc:348] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.312283 8508 raft_consensus.cc:374] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:35.312414 8508 raft_consensus.cc:729] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Initialized, Role: FOLLOWER
I20260928 05:11:35.312649 8508 consensus_queue.cc:260] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.313238 8508 ts_tablet_manager.cc:1428] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:35.421578 8561 raft_consensus.cc:482] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:35.421774 8561 raft_consensus.cc:504] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.422540 8561 leader_election.cc:288] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243), 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:35.423039 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d14022a6244709b787787e265b66dc" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895" is_pre_election: true
I20260928 05:11:35.423187 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d14022a6244709b787787e265b66dc" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02b3a623764b969aa5362ca820004f" is_pre_election: true
I20260928 05:11:35.423295 8298 raft_consensus.cc:2454] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 0.
I20260928 05:11:35.423405 8438 raft_consensus.cc:2454] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 0.
I20260928 05:11:35.423743 8087 leader_election.cc:302] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:35.424029 8561 raft_consensus.cc:2781] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:35.424165 8561 raft_consensus.cc:482] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:35.424263 8561 raft_consensus.cc:3037] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:35.427546 8561 raft_consensus.cc:504] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.428177 8561 leader_election.cc:288] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 election: Requested vote from peers ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243), 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:35.428614 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d14022a6244709b787787e265b66dc" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02b3a623764b969aa5362ca820004f"
I20260928 05:11:35.428660 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d14022a6244709b787787e265b66dc" candidate_uuid: "372ebd49ce994682a3e179a6f9c90f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3519d408241ea95f1c840a2aa8895"
I20260928 05:11:35.428802 8438 raft_consensus.cc:3037] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:35.428874 8298 raft_consensus.cc:3037] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:35.431125 8438 raft_consensus.cc:2454] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 1.
I20260928 05:11:35.431533 8088 leader_election.cc:302] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, 9f02b3a623764b969aa5362ca820004f; no voters:
I20260928 05:11:35.431788 8561 raft_consensus.cc:2781] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:35.431825 8298 raft_consensus.cc:2454] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372ebd49ce994682a3e179a6f9c90f3d in term 1.
I20260928 05:11:35.431950 8561 raft_consensus.cc:686] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 1 LEADER]: Becoming Leader. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Running, Role: LEADER
I20260928 05:11:35.432227 8561 consensus_queue.cc:237] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.435074 7999 catalog_manager.cc:5495] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "372ebd49ce994682a3e179a6f9c90f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } health_report { overall_health: UNKNOWN } } }
I20260928 05:11:35.454619 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.448923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.0.97.1:44877)
0928 05:11:35.449120 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.449127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.449177 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:11:35.449425 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.449431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.449536 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.449634 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.449938 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.451082 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.452464 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.453285 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.453431 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.453820 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.453945 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.454033 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.454208 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.454286 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:11:35.457676 8298 raft_consensus.cc:1261] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Refusing update from remote peer 372ebd49ce994682a3e179a6f9c90f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.457762 8438 raft_consensus.cc:1261] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Refusing update from remote peer 372ebd49ce994682a3e179a6f9c90f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.458364 8561 consensus_queue.cc:1035] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.458704 8514 consensus_queue.cc:1035] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.474112 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:35.474288 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:35.477243 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Sending DeleteTablet for 3 replicas of tablet 93d14022a6244709b787787e265b66dc
I20260928 05:11:35.478150 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d14022a6244709b787787e265b66dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49076
I20260928 05:11:35.478272 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d14022a6244709b787787e265b66dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58658
I20260928 05:11:35.478322 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d14022a6244709b787787e265b66dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35978
I20260928 05:11:35.478416 8575 tablet_replica.cc:330] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: stopping tablet replica
I20260928 05:11:35.478585 8576 tablet_replica.cc:330] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: stopping tablet replica
I20260928 05:11:35.478578 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d14022a6244709b787787e265b66dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58658
I20260928 05:11:35.478595 8575 raft_consensus.cc:2229] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:35.478770 8574 tablet_replica.cc:330] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: stopping tablet replica
I20260928 05:11:35.478783 8575 raft_consensus.cc:2258] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.478768 8576 raft_consensus.cc:2229] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:35.478930 8574 raft_consensus.cc:2229] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:35.479128 8574 raft_consensus.cc:2258] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.479231 8576 raft_consensus.cc:2258] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.480264 8574 ts_tablet_manager.cc:1905] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.480278 8575 ts_tablet_manager.cc:1905] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.480422 8576 ts_tablet_manager.cc:1905] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.486588 8575 ts_tablet_manager.cc:1918] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:35.486652 8574 ts_tablet_manager.cc:1918] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:35.486720 8575 log.cc:1196] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/93d14022a6244709b787787e265b66dc
I20260928 05:11:35.486790 8574 log.cc:1196] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/93d14022a6244709b787787e265b66dc
I20260928 05:11:35.487124 8575 ts_tablet_manager.cc:1939] T 93d14022a6244709b787787e265b66dc P ebe3519d408241ea95f1c840a2aa8895: Deleting consensus metadata
I20260928 05:11:35.487174 8574 ts_tablet_manager.cc:1939] T 93d14022a6244709b787787e265b66dc P 9f02b3a623764b969aa5362ca820004f: Deleting consensus metadata
I20260928 05:11:35.488099 7984 catalog_manager.cc:4841] TS ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243): tablet 93d14022a6244709b787787e265b66dc (table test-table [id=734ef0c6bf164d9f8edd00b879504309]) successfully deleted
I20260928 05:11:35.488166 7985 catalog_manager.cc:4841] TS 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979): tablet 93d14022a6244709b787787e265b66dc (table test-table [id=734ef0c6bf164d9f8edd00b879504309]) successfully deleted
I20260928 05:11:35.488168 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 05:11:35.488633 8576 ts_tablet_manager.cc:1918] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:11:35.488785 8576 log.cc:1196] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/93d14022a6244709b787787e265b66dc
W20260928 05:11:35.489012 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:11:35.489250 8576 ts_tablet_manager.cc:1939] T 93d14022a6244709b787787e265b66dc P 372ebd49ce994682a3e179a6f9c90f3d: Deleting consensus metadata
I20260928 05:11:35.490212 7984 catalog_manager.cc:4841] TS 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877): tablet 93d14022a6244709b787787e265b66dc (table test-table [id=734ef0c6bf164d9f8edd00b879504309]) successfully deleted
W20260928 05:11:35.490929 7985 catalog_manager.cc:4808] TS 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979): delete failed for tablet 93d14022a6244709b787787e265b66dc because the tablet was not found. No further retry: Not found: Tablet not found: 93d14022a6244709b787787e265b66dc
I20260928 05:11:35.495982 8136 tablet_service.cc:1467] Processing CreateTablet for tablet f6b4ddcc6d374e56b4008852b2c8ba4f (DEFAULT_TABLE table=test-table [id=2a16a7ca61a44510b491e8e2a0341b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:35.496034 8274 tablet_service.cc:1467] Processing CreateTablet for tablet f6b4ddcc6d374e56b4008852b2c8ba4f (DEFAULT_TABLE table=test-table [id=2a16a7ca61a44510b491e8e2a0341b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:35.496374 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b4ddcc6d374e56b4008852b2c8ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:35.496385 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b4ddcc6d374e56b4008852b2c8ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:35.496366 8414 tablet_service.cc:1467] Processing CreateTablet for tablet f6b4ddcc6d374e56b4008852b2c8ba4f (DEFAULT_TABLE table=test-table [id=2a16a7ca61a44510b491e8e2a0341b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:11:35.496755 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b4ddcc6d374e56b4008852b2c8ba4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:35.502854 8510 tablet_bootstrap.cc:492] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Bootstrap starting.
I20260928 05:11:35.503427 8508 tablet_bootstrap.cc:492] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Bootstrap starting.
I20260928 05:11:35.504256 8510 tablet_bootstrap.cc:654] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:35.504856 8508 tablet_bootstrap.cc:654] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:35.505101 8509 tablet_bootstrap.cc:492] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Bootstrap starting.
I20260928 05:11:35.506618 8509 tablet_bootstrap.cc:654] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:35.506800 8508 tablet_bootstrap.cc:492] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: No bootstrap required, opened a new log
I20260928 05:11:35.506948 8508 ts_tablet_manager.cc:1397] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 05:11:35.507589 8508 raft_consensus.cc:348] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.507828 8508 raft_consensus.cc:374] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:35.507894 8508 raft_consensus.cc:729] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ebd49ce994682a3e179a6f9c90f3d, State: Initialized, Role: FOLLOWER
I20260928 05:11:35.508096 8508 consensus_queue.cc:260] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.508947 8509 tablet_bootstrap.cc:492] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: No bootstrap required, opened a new log
I20260928 05:11:35.509122 8509 ts_tablet_manager.cc:1397] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 05:11:35.509940 8509 raft_consensus.cc:348] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.510185 8509 raft_consensus.cc:374] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:35.510277 8509 raft_consensus.cc:729] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f02b3a623764b969aa5362ca820004f, State: Initialized, Role: FOLLOWER
I20260928 05:11:35.510515 8509 consensus_queue.cc:260] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.511051 8509 ts_tablet_manager.cc:1428] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 05:11:35.511257 8508 ts_tablet_manager.cc:1428] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 05:11:35.518599 8510 tablet_bootstrap.cc:492] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: No bootstrap required, opened a new log
I20260928 05:11:35.518779 8510 ts_tablet_manager.cc:1397] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20260928 05:11:35.519562 8510 raft_consensus.cc:348] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.519827 8510 raft_consensus.cc:374] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:35.519912 8510 raft_consensus.cc:729] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3519d408241ea95f1c840a2aa8895, State: Initialized, Role: FOLLOWER
I20260928 05:11:35.520136 8510 consensus_queue.cc:260] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.520731 8510 ts_tablet_manager.cc:1428] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 05:11:35.678715 8515 raft_consensus.cc:482] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:11:35.678965 8515 raft_consensus.cc:504] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.680401 8515 leader_election.cc:288] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877), 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:35.690946 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.681192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58963 (local address 127.0.97.1:44877)
0928 05:11:35.681380 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.681386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.681408 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:11:35.681653 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.681663 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.681760 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.681858 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:35.682691 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.684265 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.687530 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.689345 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.689474 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.689628 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.690358 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.690554 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:11:35.691737 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.680602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44877 (local address 127.0.97.2:58963)
0928 05:11:35.681235 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.681273 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.681433 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.682116 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.682125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.682158 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:35.682459 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.682474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.684515 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.684523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.687279 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.687300 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.689849 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.689858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.690128 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.690180 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.691248 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.691398 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":809,"threads_started":1}
I20260928 05:11:35.692386 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b4ddcc6d374e56b4008852b2c8ba4f" candidate_uuid: "ebe3519d408241ea95f1c840a2aa8895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ebd49ce994682a3e179a6f9c90f3d" is_pre_election: true
I20260928 05:11:35.692672 8161 raft_consensus.cc:2454] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe3519d408241ea95f1c840a2aa8895 in term 0.
I20260928 05:11:35.693256 8227 leader_election.cc:302] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:35.693663 8515 raft_consensus.cc:2781] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:11:35.693796 8515 raft_consensus.cc:482] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:11:35.693886 8515 raft_consensus.cc:3037] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:35.697438 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.680622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35979 (local address 127.0.97.2:40111)
0928 05:11:35.691772 (+ 11150us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.691787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.691934 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.692345 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.692350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.692379 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:11:35.692706 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.692717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.694131 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.694136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.695602 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.695614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.696705 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.696710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.696871 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.696893 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.696989 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.697107 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11038,"thread_start_us":10860,"threads_started":1}
I20260928 05:11:35.697335 8515 raft_consensus.cc:504] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.697947 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.680954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40111 (local address 127.0.97.3:35979)
0928 05:11:35.691299 (+ 10345us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.691315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.692031 (+ 716us) server_negotiation.cc:407] Connection header received
0928 05:11:35.692107 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.692111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.692213 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.692314 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:11:35.692920 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.693965 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.695755 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.696490 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.696637 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.696744 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.697023 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.697155 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10225,"thread_start_us":10165,"threads_started":1}
I20260928 05:11:35.698680 8161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b4ddcc6d374e56b4008852b2c8ba4f" candidate_uuid: "ebe3519d408241ea95f1c840a2aa8895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ebd49ce994682a3e179a6f9c90f3d"
I20260928 05:11:35.698860 8515 leader_election.cc:288] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [CANDIDATE]: Term 1 election: Requested vote from peers 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877), 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979)
I20260928 05:11:35.698915 8161 raft_consensus.cc:3037] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:35.698869 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b4ddcc6d374e56b4008852b2c8ba4f" candidate_uuid: "ebe3519d408241ea95f1c840a2aa8895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02b3a623764b969aa5362ca820004f" is_pre_election: true
I20260928 05:11:35.699069 8438 raft_consensus.cc:2454] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe3519d408241ea95f1c840a2aa8895 in term 0.
I20260928 05:11:35.699421 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b4ddcc6d374e56b4008852b2c8ba4f" candidate_uuid: "ebe3519d408241ea95f1c840a2aa8895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f02b3a623764b969aa5362ca820004f"
I20260928 05:11:35.699574 8438 raft_consensus.cc:3037] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:35.701323 8161 raft_consensus.cc:2454] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe3519d408241ea95f1c840a2aa8895 in term 1.
I20260928 05:11:35.701694 8227 leader_election.cc:302] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372ebd49ce994682a3e179a6f9c90f3d, ebe3519d408241ea95f1c840a2aa8895; no voters:
I20260928 05:11:35.701964 8515 raft_consensus.cc:2781] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:35.702272 8438 raft_consensus.cc:2454] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe3519d408241ea95f1c840a2aa8895 in term 1.
I20260928 05:11:35.702431 8515 raft_consensus.cc:686] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 1 LEADER]: Becoming Leader. State: Replica: ebe3519d408241ea95f1c840a2aa8895, State: Running, Role: LEADER
I20260928 05:11:35.702855 8515 consensus_queue.cc:237] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } }
I20260928 05:11:35.706103 7999 catalog_manager.cc:5495] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ebe3519d408241ea95f1c840a2aa8895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebe3519d408241ea95f1c840a2aa8895" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 } health_report { overall_health: UNKNOWN } } }
I20260928 05:11:35.762838 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.757012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.97.2:40243)
0928 05:11:35.757734 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.757739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.757757 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:11:35.757837 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.757840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.757906 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.758075 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.758574 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.759333 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.760609 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.761019 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.761149 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.761593 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.762181 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.762365 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.762412 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.762496 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":504,"threads_started":1}
I20260928 05:11:35.778905 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.772670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55291 (local address 127.0.97.1:44877)
0928 05:11:35.772863 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.772873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.773392 (+ 519us) server_negotiation.cc:407] Connection header received
0928 05:11:35.773594 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.773598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.773682 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.773755 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.774362 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.775159 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.777136 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.777613 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.777703 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.778221 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.778337 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.778403 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.778620 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.778685 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:11:35.779096 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.772585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44877 (local address 127.0.97.2:55291)
0928 05:11:35.773285 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.773305 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.773448 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.773807 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.773813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.773826 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:11:35.774197 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.774209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.775302 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.775308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.777006 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.777021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.777799 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.777807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.777976 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.778056 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.778435 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.778448 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.778553 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.778692 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":332,"threads_started":1}
I20260928 05:11:35.787524 8438 raft_consensus.cc:1261] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Refusing update from remote peer ebe3519d408241ea95f1c840a2aa8895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.787621 8161 raft_consensus.cc:1261] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Refusing update from remote peer ebe3519d408241ea95f1c840a2aa8895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:11:35.788194 8602 consensus_queue.cc:1035] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f02b3a623764b969aa5362ca820004f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.788615 8515 consensus_queue.cc:1035] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372ebd49ce994682a3e179a6f9c90f3d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:11:35.794627 8557 mvcc.cc:205] Tried to move back new op lower bound from 7334184123540037632 to 7334184123202260992. Current Snapshot: MvccSnapshot[applied={T|T < 7334184123540037632}]
I20260928 05:11:35.822106 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.815795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55779 (local address 127.0.97.2:40243)
0928 05:11:35.815989 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.815997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.816338 (+ 341us) server_negotiation.cc:407] Connection header received
0928 05:11:35.816596 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.816603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.816684 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.816790 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:11:35.817494 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.818229 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.820291 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.820908 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.821018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.821475 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.821562 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.821626 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.821783 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.821840 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:11:35.822325 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.815663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40243 (local address 127.0.97.1:55779)
0928 05:11:35.816244 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:35.816259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:35.816395 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:35.816970 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:35.816974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:35.816987 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 05:11:35.817319 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.817338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.818362 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.818368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.820135 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:35.820149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.821103 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.821110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:35.821292 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.821341 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 05:11:35.821657 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 05:11:35.821667 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:11:35.821771 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 05:11:35.821907 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":354,"threads_started":1}
I20260928 05:11:35.857167 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:35.857466 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:11:35.860265 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60584990d27348af81b442b71dcd92b0: Sending DeleteTablet for 3 replicas of tablet f6b4ddcc6d374e56b4008852b2c8ba4f
I20260928 05:11:35.861407 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b4ddcc6d374e56b4008852b2c8ba4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49076
I20260928 05:11:35.861402 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b4ddcc6d374e56b4008852b2c8ba4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58658
I20260928 05:11:35.861671 8574 tablet_replica.cc:330] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: stopping tablet replica
I20260928 05:11:35.861681 8575 tablet_replica.cc:330] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: stopping tablet replica
I20260928 05:11:35.861835 8574 raft_consensus.cc:2229] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:35.861847 8575 raft_consensus.cc:2229] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:11:35.862031 8574 raft_consensus.cc:2258] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.862331 8575 raft_consensus.cc:2258] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.862840 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b4ddcc6d374e56b4008852b2c8ba4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:11:35 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35978
I20260928 05:11:35.863184 8576 tablet_replica.cc:330] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: stopping tablet replica
I20260928 05:11:35.863353 8576 raft_consensus.cc:2229] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:11:35.863508 8575 ts_tablet_manager.cc:1905] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.863549 8576 raft_consensus.cc:2258] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:11:35.864653 8574 ts_tablet_manager.cc:1905] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.864799 8576 ts_tablet_manager.cc:1905] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:11:35.870136 8575 ts_tablet_manager.cc:1918] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:11:35.870325 8575 log.cc:1196] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/f6b4ddcc6d374e56b4008852b2c8ba4f
I20260928 05:11:35.870828 8575 ts_tablet_manager.cc:1939] T f6b4ddcc6d374e56b4008852b2c8ba4f P ebe3519d408241ea95f1c840a2aa8895: Deleting consensus metadata
I20260928 05:11:35.871868 8576 ts_tablet_manager.cc:1918] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:11:35.872001 8576 log.cc:1196] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/f6b4ddcc6d374e56b4008852b2c8ba4f
I20260928 05:11:35.872375 7984 catalog_manager.cc:4841] TS ebe3519d408241ea95f1c840a2aa8895 (127.0.97.2:40243): tablet f6b4ddcc6d374e56b4008852b2c8ba4f (table test-table [id=2a16a7ca61a44510b491e8e2a0341b93]) successfully deleted
I20260928 05:11:35.872447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 8070
I20260928 05:11:35.872475 8576 ts_tablet_manager.cc:1939] T f6b4ddcc6d374e56b4008852b2c8ba4f P 372ebd49ce994682a3e179a6f9c90f3d: Deleting consensus metadata
I20260928 05:11:35.873704 8574 ts_tablet_manager.cc:1918] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 05:11:35.873852 8574 log.cc:1196] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/f6b4ddcc6d374e56b4008852b2c8ba4f
I20260928 05:11:35.874238 8574 ts_tablet_manager.cc:1939] T f6b4ddcc6d374e56b4008852b2c8ba4f P 9f02b3a623764b969aa5362ca820004f: Deleting consensus metadata
I20260928 05:11:35.875339 7985 catalog_manager.cc:4841] TS 9f02b3a623764b969aa5362ca820004f (127.0.97.3:35979): tablet f6b4ddcc6d374e56b4008852b2c8ba4f (table test-table [id=2a16a7ca61a44510b491e8e2a0341b93]) successfully deleted
I20260928 05:11:35.875514 7984 catalog_manager.cc:4841] TS 372ebd49ce994682a3e179a6f9c90f3d (127.0.97.1:44877): tablet f6b4ddcc6d374e56b4008852b2c8ba4f (table test-table [id=2a16a7ca61a44510b491e8e2a0341b93]) successfully deleted
Sep 28 05:11:35 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 05:11:35.893828 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:35.874878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.0.97.1:44877)
0928 05:11:35.875395 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:35.875401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:35.879178 (+ 3777us) server_negotiation.cc:407] Connection header received
0928 05:11:35.879403 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:35.879409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:35.879503 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:35.879613 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:35.880753 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.881435 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.882682 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:35.883159 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:35.883315 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:35.887353 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:35.887369 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:35.887384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:35.887416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:35.890965 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:35.891624 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:35.891640 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:35.891644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:35.891759 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:35.892141 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:35.892146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:35.892149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:35.892532 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:35.892889 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:35.893176 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 05:11:35.893438 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260928 05:11:35.894629 8201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:36.053470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 8070
I20260928 05:11:36.079808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 8210
Sep 28 05:11:36 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 05:11:36.096849 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:36.080594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.0.97.2:40243)
0928 05:11:36.080797 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:36.080804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:36.080828 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:36.080977 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:36.080983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:36.081063 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:36.081175 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:36.082383 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:36.083544 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:36.084777 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:36.085511 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:36.085652 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:36.089658 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:36.089671 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:36.089690 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:36.089731 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:36.093765 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:36.094690 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:36.094703 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:36.094708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:36.094902 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:36.095251 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:36.095265 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:36.095269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:36.095697 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:36.096024 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:36.096202 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 05:11:36.096438 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:11:36.097575 8340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:36.224113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 8210
I20260928 05:11:36.244145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 8349
Sep 28 05:11:36 dist-test-slave-6718 krb5kdc[7954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 05:11:36.258934 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:36.244853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.97.3:35979)
0928 05:11:36.245321 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:36.245325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:36.245341 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:11:36.245393 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:36.245396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:36.245458 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:36.245553 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:36.246798 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:36.247461 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:36.248613 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:36.248972 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:36.249061 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:36.253356 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:36.253372 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:36.253389 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:36.253421 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:36.256129 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:36.256756 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:36.256765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:36.256767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:36.256904 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:36.257234 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:36.257241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:36.257246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:36.257623 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:36.257822 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:36.258156 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 05:11:36.258374 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":263,"threads_started":1}
I20260928 05:11:36.259546 8479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:36.380961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 8349
I20260928 05:11:36.402148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 7967
I20260928 05:11:36.403368 8039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:36.516659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 7967
2026-09-28T05:11:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5736 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:36 dist-test-slave-6718 krb5kdc[8622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:36 dist-test-slave-6718 krb5kdc[8622](info): set up 2 sockets
Sep 28 05:11:36 dist-test-slave-6718 krb5kdc[8622](info): commencing operation
krb5kdc: starting...
W20260928 05:11:37.619522 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:37 dist-test-slave-6718 krb5kdc[8622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:11:37.838454 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46023
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:38.159363 8635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:38.159636 8635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:38.159734 8635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:38.166952 8635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:38.167061 8635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:38.167106 8635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:38.167196 8635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:38.167244 8635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:38.177320 8635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:38.178679 8635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:38.179984 8635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:38.191376 8640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:38.191422 8641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:38.191331 8643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:38.195916 8635 server_base.cc:1034] running on GCE node
I20260928 05:11:38.196630 8635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:38.198045 8635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:38.199249 8635 hybrid_clock.cc:648] HybridClock initialized: now 1790572298199201 us; error 58 us; skew 500 ppm
I20260928 05:11:38.199554 8635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T05:11:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1748 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:11:38 dist-test-slave-6718 krb5kdc[8649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:11:38 dist-test-slave-6718 krb5kdc[8649](info): set up 2 sockets
Sep 28 05:11:38 dist-test-slave-6718 krb5kdc[8649](info): commencing operation
krb5kdc: starting...
W20260928 05:11:39.343801 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:11:39 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:11:39.566691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:39.883814 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:39.884160 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:39.884332 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:39.891662 8662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:39.891832 8662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:39.891945 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:39.892050 8662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:39.892139 8662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:39.902212 8662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:39.903626 8662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:39.904983 8662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:39.916805 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:39.916890 8667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:39.916852 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:39.917618 8662 server_base.cc:1034] running on GCE node
I20260928 05:11:39.918398 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:39.919781 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:39.921010 8662 hybrid_clock.cc:648] HybridClock initialized: now 1790572299920964 us; error 58 us; skew 500 ppm
I20260928 05:11:39.921345 8662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:39 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:39.929399 8662 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:11:39.931016 8662 webserver.cc:460] Webserver started at http://127.0.97.62:40793/ using document root <none> and password file <none>
I20260928 05:11:39.931758 8662 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:39.931907 8662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:39.932245 8662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:39.935370 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "f324caca3419487d8c99934fc1097de9"
format_stamp: "Formatted at 2026-09-28 05:11:39 on dist-test-slave-6718"
server_key: "1cbe8cdd57b6e9b1a8fe98bd9066e6ec"
server_key_iv: "9243eda3d8ce5c0d253ce133ce284068"
server_key_version: "encryptionkey@0"
I20260928 05:11:39.936187 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "f324caca3419487d8c99934fc1097de9"
format_stamp: "Formatted at 2026-09-28 05:11:39 on dist-test-slave-6718"
server_key: "1cbe8cdd57b6e9b1a8fe98bd9066e6ec"
server_key_iv: "9243eda3d8ce5c0d253ce133ce284068"
server_key_version: "encryptionkey@0"
I20260928 05:11:39.941504 8662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:11:39.945117 8677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:39.946467 8662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:11:39.946679 8662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "f324caca3419487d8c99934fc1097de9"
format_stamp: "Formatted at 2026-09-28 05:11:39 on dist-test-slave-6718"
server_key: "1cbe8cdd57b6e9b1a8fe98bd9066e6ec"
server_key_iv: "9243eda3d8ce5c0d253ce133ce284068"
server_key_version: "encryptionkey@0"
I20260928 05:11:39.946882 8662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:39.973608 8662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:39.978107 8662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:39.978410 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:39.997009 8662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41229
I20260928 05:11:39.997040 8728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41229 every 8 connection(s)
I20260928 05:11:39.998481 8662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:40.002228 8729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:40.007799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 8662
I20260928 05:11:40.008023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:40.008781 388 external_mini_cluster.cc:1428] Setting key 3694a6f77d9cc39b82d4b297ba4cccc6
I20260928 05:11:40.013926 8729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Bootstrap starting.
I20260928 05:11:40.017061 8729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:40.018115 8729 log.cc:824] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:40.021008 8729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: No bootstrap required, opened a new log
Sep 28 05:11:40 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:40.029881 8729 raft_consensus.cc:348] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:40.030364 8729 raft_consensus.cc:374] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:40.030511 8729 raft_consensus.cc:729] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f324caca3419487d8c99934fc1097de9, State: Initialized, Role: FOLLOWER
I20260928 05:11:40.031203 8729 consensus_queue.cc:260] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:40.031436 8729 raft_consensus.cc:388] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:40.031569 8729 raft_consensus.cc:482] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:40.031757 8729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:40.031801 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:40.010991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.97.62:41229)
0928 05:11:40.011791 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:40.011802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:40.011845 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:11:40.013503 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:40.013526 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:40.013939 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:40.014392 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:40.015857 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:40.017167 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:40.018512 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:40.019682 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:40.019888 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:40.024145 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:40.024161 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:40.024177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:40.024214 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:40.027446 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:40.028253 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:40.028260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:40.028267 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:40.028432 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:40.028807 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:40.028810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:40.028811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:40.029335 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:40.029543 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:40.030011 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 05:11:40.030621 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":348,"threads_started":1}
I20260928 05:11:40.034032 8729 raft_consensus.cc:504] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:40.034593 8729 leader_election.cc:302] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f324caca3419487d8c99934fc1097de9; no voters:
I20260928 05:11:40.035182 8729 leader_election.cc:288] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:40.035286 8734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:40.035986 8734 raft_consensus.cc:686] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 LEADER]: Becoming Leader. State: Replica: f324caca3419487d8c99934fc1097de9, State: Running, Role: LEADER
I20260928 05:11:40.036576 8734 consensus_queue.cc:237] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:40.037253 8729 sys_catalog.cc:564] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:40.039173 8735 sys_catalog.cc:455] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f324caca3419487d8c99934fc1097de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } } }
I20260928 05:11:40.039700 8735 sys_catalog.cc:458] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:40.039987 8736 sys_catalog.cc:455] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f324caca3419487d8c99934fc1097de9. Latest consensus state: current_term: 1 leader_uuid: "f324caca3419487d8c99934fc1097de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } } }
I20260928 05:11:40.040304 8736 sys_catalog.cc:458] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:40.047407 8742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:40.053059 8742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:40.062701 8742 catalog_manager.cc:1347] Generated new cluster ID: 0c7d39531d1e471d89e0d9a45a083b78
I20260928 05:11:40.062893 8742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:40.075266 8742 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:40.077086 8742 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:40.084839 8742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Generated new TSK 0
I20260928 05:11:40.085786 8742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:11:40.150769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41229
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:40.472759 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:40.473084 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:40.473249 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:40.480405 8755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:40.480562 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:40.480728 8755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:40.490346 8755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:40.491952 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:40.493335 8755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:40.506558 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:40.506595 8761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:40.506767 8763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:40.507369 8755 server_base.cc:1034] running on GCE node
I20260928 05:11:40.507913 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:40.508716 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:40.509912 8755 hybrid_clock.cc:648] HybridClock initialized: now 1790572300509879 us; error 31 us; skew 500 ppm
I20260928 05:11:40.510219 8755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:40 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:40.513384 8755 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:11:40.514787 8755 webserver.cc:460] Webserver started at http://127.0.97.1:44697/ using document root <none> and password file <none>
I20260928 05:11:40.515437 8755 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:40.515576 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:40.515873 8755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:40.518652 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "8bfc91fb0f954d89bb05f012de5fa481"
format_stamp: "Formatted at 2026-09-28 05:11:40 on dist-test-slave-6718"
server_key: "c06dfba68263752bd5128fdaf4aedf84"
server_key_iv: "575f337f87e14f9ae8b1c399a69354a7"
server_key_version: "encryptionkey@0"
I20260928 05:11:40.519410 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8bfc91fb0f954d89bb05f012de5fa481"
format_stamp: "Formatted at 2026-09-28 05:11:40 on dist-test-slave-6718"
server_key: "c06dfba68263752bd5128fdaf4aedf84"
server_key_iv: "575f337f87e14f9ae8b1c399a69354a7"
server_key_version: "encryptionkey@0"
I20260928 05:11:40.524178 8755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 05:11:40.527856 8770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:40.529132 8755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:11:40.529336 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "8bfc91fb0f954d89bb05f012de5fa481"
format_stamp: "Formatted at 2026-09-28 05:11:40 on dist-test-slave-6718"
server_key: "c06dfba68263752bd5128fdaf4aedf84"
server_key_iv: "575f337f87e14f9ae8b1c399a69354a7"
server_key_version: "encryptionkey@0"
I20260928 05:11:40.529551 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:40.552656 8755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:40.557214 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:40.557509 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:40.558460 8755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:40.559765 8755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:40.559873 8755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:40.560017 8755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:40.560091 8755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:40.593712 8755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34739
I20260928 05:11:40.593745 8882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34739 every 8 connection(s)
I20260928 05:11:40.595045 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:40.603865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 8755
I20260928 05:11:40.604086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:11:40.604661 388 external_mini_cluster.cc:1428] Setting key ea47d18ca8495f01ff38a5f0de84f5ae
Sep 28 05:11:40 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:40.616276 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:40.597554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34729 (local address 127.0.97.62:41229)
0928 05:11:40.598011 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:40.598016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:40.599115 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 05:11:40.600924 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:40.600931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:40.601018 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:40.601133 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:40.602626 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:40.603343 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:40.605061 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:40.605603 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:40.605755 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:40.609930 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:40.609939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:40.609947 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:40.609989 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:40.613218 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:40.613943 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:40.613954 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:40.613956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:40.614063 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:40.614610 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:40.614614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:40.614616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:40.614866 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:40.614989 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:40.615839 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 05:11:40.616006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260928 05:11:40.617754 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:40.598134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:34729)
0928 05:11:40.598943 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:40.598990 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:40.600651 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:40.601394 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:40.601401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:40.601902 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:40.602437 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:40.602458 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:40.603517 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:40.603521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:40.604918 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:40.604929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:40.605830 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:40.605838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:40.606104 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:40.606962 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:40.606985 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:40.609661 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:40.613368 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:40.613377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:40.613394 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:40.613815 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:40.614169 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:40.614177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:40.614183 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:40.614418 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:40.614990 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:40.615000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:40.615447 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 05:11:40.615738 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 05:11:40.616317 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":351,"threads_started":1}
I20260928 05:11:40.619393 8883 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41229
I20260928 05:11:40.619692 8883 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:40.620383 8883 heartbeater.cc:511] Master 127.0.97.62:41229 requested a full tablet report, sending...
I20260928 05:11:40.622377 8694 ts_manager.cc:194] Registered new tserver with Master: 8bfc91fb0f954d89bb05f012de5fa481 (127.0.97.1:34739)
I20260928 05:11:40.623905 8694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34729
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:11:40.653561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41229
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:40.972321 8889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:40.972633 8889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:40.972795 8889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:40.979800 8889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:40.979945 8889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:40.980106 8889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:40.989573 8889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:40.990948 8889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:40.992218 8889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:41.005429 8894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:41.005451 8895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:41.006350 8889 server_base.cc:1034] running on GCE node
W20260928 05:11:41.007054 8897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:41.007756 8889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:41.008731 8889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:41.009995 8889 hybrid_clock.cc:648] HybridClock initialized: now 1790572301009941 us; error 46 us; skew 500 ppm
I20260928 05:11:41.010320 8889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:41.013430 8889 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:11:41.014930 8889 webserver.cc:460] Webserver started at http://127.0.97.2:39773/ using document root <none> and password file <none>
I20260928 05:11:41.015579 8889 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:41.015700 8889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:41.016001 8889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:41.018914 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "b8b64df4d8da4df4af3e0d7ec80a358a"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "91d0dbb8f12e9cc5036b39b5f213f357"
server_key_iv: "4a567beef6c994149883b6d592147c33"
server_key_version: "encryptionkey@0"
I20260928 05:11:41.019668 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b8b64df4d8da4df4af3e0d7ec80a358a"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "91d0dbb8f12e9cc5036b39b5f213f357"
server_key_iv: "4a567beef6c994149883b6d592147c33"
server_key_version: "encryptionkey@0"
I20260928 05:11:41.024580 8889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:11:41.027944 8904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:41.029188 8889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:11:41.029348 8889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "b8b64df4d8da4df4af3e0d7ec80a358a"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "91d0dbb8f12e9cc5036b39b5f213f357"
server_key_iv: "4a567beef6c994149883b6d592147c33"
server_key_version: "encryptionkey@0"
I20260928 05:11:41.029539 8889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:41.047326 8889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:41.051646 8889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:41.051905 8889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:41.052793 8889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:41.054023 8889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:41.054121 8889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:41.054263 8889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:41.054330 8889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:41.085981 8889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37721
I20260928 05:11:41.086017 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37721 every 8 connection(s)
I20260928 05:11:41.087340 8889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:41.092830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 8889
I20260928 05:11:41.093014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:11:41.093477 388 external_mini_cluster.cc:1428] Setting key bbfaf192db04b6ef2941139fd839d97d
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:41.109040 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.089772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45667 (local address 127.0.97.62:41229)
0928 05:11:41.089986 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:41.089991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:41.091153 (+ 1162us) server_negotiation.cc:407] Connection header received
0928 05:11:41.092889 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:41.092898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:41.092989 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:41.093112 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:41.094628 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.095571 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.097382 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.098143 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.098292 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.102281 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:41.102287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:41.102292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:41.102334 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:41.105973 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.106795 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.106803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.106810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.106956 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.107429 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.107433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.107435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.107671 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:41.107826 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:41.108492 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 05:11:41.108702 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:11:41.109984 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.090249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:45667)
0928 05:11:41.090985 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:41.091020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:41.092620 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:41.093375 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:41.093383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:41.093919 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:41.094447 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:41.094463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.095778 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.095788 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:41.097245 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:41.097256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.098410 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.098415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:41.098697 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.099402 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:41.099428 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:41.102014 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:41.106123 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:41.106130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:41.106142 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:41.106660 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:41.107089 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:41.107093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:41.107095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:41.107266 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:41.107809 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:41.107816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:41.108142 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 05:11:41.108395 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 05:11:41.108914 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":328,"threads_started":1}
I20260928 05:11:41.111548 9017 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41229
I20260928 05:11:41.111840 9017 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:41.112545 9017 heartbeater.cc:511] Master 127.0.97.62:41229 requested a full tablet report, sending...
I20260928 05:11:41.114010 8694 ts_manager.cc:194] Registered new tserver with Master: b8b64df4d8da4df4af3e0d7ec80a358a (127.0.97.2:37721)
I20260928 05:11:41.114867 8694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45667
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:11:41.143308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41229
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:41.458482 9022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:41.458755 9022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:41.458905 9022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:41.465865 9022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:41.465958 9022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:41.466161 9022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:41.475683 9022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:41.476841 9022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:41.478094 9022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:41.491119 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:41.491056 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:41.491073 9030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:41.491983 9022 server_base.cc:1034] running on GCE node
I20260928 05:11:41.492540 9022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:41.493441 9022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:41.494619 9022 hybrid_clock.cc:648] HybridClock initialized: now 1790572301494564 us; error 47 us; skew 500 ppm
I20260928 05:11:41.494912 9022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:41.497800 9022 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:11:41.499238 9022 webserver.cc:460] Webserver started at http://127.0.97.3:38219/ using document root <none> and password file <none>
I20260928 05:11:41.499878 9022 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:41.500020 9022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:41.500314 9022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:41.503587 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "cef05e4e37f94a128213ef79aa1a61b4"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "db5a4e4f031b35c8c8f28ae4022e9e9d"
server_key_iv: "d5396c62827bc68d528000d20a4ebcd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:41.504326 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cef05e4e37f94a128213ef79aa1a61b4"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "db5a4e4f031b35c8c8f28ae4022e9e9d"
server_key_iv: "d5396c62827bc68d528000d20a4ebcd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:41.509155 9022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 05:11:41.512365 9037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:41.513783 9022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:11:41.513974 9022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "cef05e4e37f94a128213ef79aa1a61b4"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "db5a4e4f031b35c8c8f28ae4022e9e9d"
server_key_iv: "d5396c62827bc68d528000d20a4ebcd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:41.514175 9022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:41.545229 9022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:41.549243 9022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:41.549526 9022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:41.550474 9022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:41.551919 9022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:41.552016 9022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:41.552169 9022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:41.552243 9022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:41.584432 9022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44321
I20260928 05:11:41.584478 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44321 every 8 connection(s)
I20260928 05:11:41.585683 9022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:41.592080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9022
I20260928 05:11:41.592280 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:11:41.592677 388 external_mini_cluster.cc:1428] Setting key f170646529311fe2e2d8a0ce2804b4b7
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:11:41.605501 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.588177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38475 (local address 127.0.97.62:41229)
0928 05:11:41.588341 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:41.588345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:41.589483 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 05:11:41.591333 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:41.591338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:41.591403 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:41.591484 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:41.593059 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.594153 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.596224 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.596626 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.596772 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.600099 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:41.600103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:41.600107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:41.600136 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:41.602673 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.603355 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.603360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.603363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.603487 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.603899 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.603904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.603908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.604155 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:41.604323 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:41.604963 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 05:11:41.605165 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:11:41.606437 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.588572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:38475)
0928 05:11:41.589327 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:41.589377 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:41.591080 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:41.591677 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:41.591685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:41.592226 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:41.592887 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:41.592905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.594305 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.594310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:41.596082 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:41.596094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.596862 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.596869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:41.597098 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.597740 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:41.597774 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:41.599897 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:11:41.602815 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:41.602823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:41.602838 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:41.603226 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:41.603594 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:11:41.603599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:11:41.603603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:11:41.603784 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:11:41.604276 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:11:41.604283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:11:41.604581 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 05:11:41.604852 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 05:11:41.605321 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":332,"threads_started":1}
I20260928 05:11:41.607959 9150 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41229
I20260928 05:11:41.608335 9150 heartbeater.cc:464] Registering TS with master...
I20260928 05:11:41.608997 9150 heartbeater.cc:511] Master 127.0.97.62:41229 requested a full tablet report, sending...
I20260928 05:11:41.610249 8694 ts_manager.cc:194] Registered new tserver with Master: cef05e4e37f94a128213ef79aa1a61b4 (127.0.97.3:44321)
I20260928 05:11:41.611116 8694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38475
I20260928 05:11:41.620368 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:11:41.626744 8883 heartbeater.cc:503] Master 127.0.97.62:41229 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 05:11:41.749949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 8755
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:11:41.765827 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.751056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.97.1:34739)
0928 05:11:41.751663 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:41.751670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:41.751699 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:11:41.751886 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:41.751893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:41.752109 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:41.752238 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:41.753497 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.754492 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.755775 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.756316 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.756414 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.760480 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:41.760492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:41.760507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:41.760540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:41.763350 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.763992 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.763998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.764000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.764112 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.764475 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.764479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.764481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.764789 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:41.765004 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:41.765223 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 05:11:41.765418 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
I20260928 05:11:41.767140 8878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:41.848292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 8755
I20260928 05:11:41.864552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 8889
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:11:41.880877 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.865449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.0.97.2:37721)
0928 05:11:41.866034 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:41.866041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:41.866066 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:11:41.866130 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:41.866137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:41.866353 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:41.866491 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:41.867880 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.868893 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.870127 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.870833 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.870938 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.875424 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:41.875436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:41.875451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:41.875491 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:41.878203 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.878943 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.878951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.878953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.879094 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.879455 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:41.879459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:41.879461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:41.879810 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:41.880036 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:41.880304 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:11:41.880532 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":328,"threads_started":1}
I20260928 05:11:41.882087 9012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:41.969897 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 8889
I20260928 05:11:41.985548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9022
Sep 28 05:11:41 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:11:42.001993 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:41.986619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.0.97.3:44321)
0928 05:11:41.987224 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:41.987230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:41.987258 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:11:41.987315 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:41.987321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:41.987529 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:41.987656 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:41.988798 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.989881 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.991157 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:41.991678 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:41.991774 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:41.995988 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:41.995998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:41.996012 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:41.996046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:41.999387 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:41.999991 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:42.000000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:42.000004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:42.000143 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:42.000480 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:42.000484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:42.000488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:42.000892 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:42.001190 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:42.001354 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 05:11:42.001597 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":303,"threads_started":1}
I20260928 05:11:42.003450 9145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:42.085521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9022
I20260928 05:11:42.101055 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 8662
I20260928 05:11:42.102243 8724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:42.205641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 8662
I20260928 05:11:42.225653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41229
--webserver_interface=127.0.97.62
--webserver_port=40793
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:11:42.536912 9167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:42.537241 9167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:42.537595 9167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:42.544902 9167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:42.545044 9167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:42.545145 9167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:42.545243 9167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:42.545325 9167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:42.555066 9167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:42.556345 9167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:42.557605 9167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:42.568910 9175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:42.568933 9172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:42.568933 9173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:42.569839 9167 server_base.cc:1034] running on GCE node
I20260928 05:11:42.570649 9167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:42.571986 9167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:42.573240 9167 hybrid_clock.cc:648] HybridClock initialized: now 1790572302573182 us; error 59 us; skew 500 ppm
I20260928 05:11:42.573632 9167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:42 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:42.576627 9167 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:11:42.578102 9167 webserver.cc:460] Webserver started at http://127.0.97.62:40793/ using document root <none> and password file <none>
I20260928 05:11:42.578847 9167 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:42.578998 9167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:42.584128 9167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:11:42.586911 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:42.588212 9167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:42.588405 9167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "f324caca3419487d8c99934fc1097de9"
format_stamp: "Formatted at 2026-09-28 05:11:39 on dist-test-slave-6718"
server_key: "1cbe8cdd57b6e9b1a8fe98bd9066e6ec"
server_key_iv: "9243eda3d8ce5c0d253ce133ce284068"
server_key_version: "encryptionkey@0"
I20260928 05:11:42.589427 9167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:42.603255 9167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:42.607321 9167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:42.607590 9167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:42.625288 9167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41229
I20260928 05:11:42.625322 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41229 every 8 connection(s)
I20260928 05:11:42.626812 9167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:42.633239 9234 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:11:42.634954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9167
I20260928 05:11:42.635833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34739
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41229
--webserver_port=44697
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:11:42.637436 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Bootstrap starting.
I20260928 05:11:42.657573 9234 log.cc:824] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:42.665742 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:11:42.666239 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Bootstrap complete.
I20260928 05:11:42.674952 9234 raft_consensus.cc:348] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:42.675707 9234 raft_consensus.cc:729] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f324caca3419487d8c99934fc1097de9, State: Initialized, Role: FOLLOWER
I20260928 05:11:42.676291 9234 consensus_queue.cc:260] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:42.676519 9234 raft_consensus.cc:388] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:42.676625 9234 raft_consensus.cc:482] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:42.676805 9234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:11:42.680344 9234 raft_consensus.cc:504] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:42.680801 9234 leader_election.cc:302] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f324caca3419487d8c99934fc1097de9; no voters:
I20260928 05:11:42.681416 9234 leader_election.cc:288] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:11:42.681532 9238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:11:42.682418 9238 raft_consensus.cc:686] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [term 2 LEADER]: Becoming Leader. State: Replica: f324caca3419487d8c99934fc1097de9, State: Running, Role: LEADER
I20260928 05:11:42.682974 9238 consensus_queue.cc:237] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } }
I20260928 05:11:42.683445 9234 sys_catalog.cc:564] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:42.685467 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f324caca3419487d8c99934fc1097de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } } }
I20260928 05:11:42.685868 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:42.694161 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f324caca3419487d8c99934fc1097de9. Latest consensus state: current_term: 2 leader_uuid: "f324caca3419487d8c99934fc1097de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f324caca3419487d8c99934fc1097de9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41229 } } }
I20260928 05:11:42.694545 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:42.695652 9244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:42.702246 9244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:42.704095 9244 catalog_manager.cc:1259] Loaded cluster ID: 0c7d39531d1e471d89e0d9a45a083b78
I20260928 05:11:42.704289 9244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:42.709651 9244 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:42.711525 9244 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f324caca3419487d8c99934fc1097de9: Loaded TSK: 0
I20260928 05:11:42.713367 9244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:11:42.984743 9236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:42.985054 9236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:42.985203 9236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:42.992457 9236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:42.992592 9236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:42.992807 9236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:11:43.002281 9236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44697
--tserver_master_addrs=127.0.97.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:43.003556 9236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:43.004802 9236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:43.017403 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:43.017423 9261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:43.018296 9263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:43.019047 9236 server_base.cc:1034] running on GCE node
I20260928 05:11:43.019753 9236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:43.020892 9236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:43.022130 9236 hybrid_clock.cc:648] HybridClock initialized: now 1790572303022063 us; error 60 us; skew 500 ppm
I20260928 05:11:43.022500 9236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572303, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:43.026494 9236 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 05:11:43.028208 9236 webserver.cc:460] Webserver started at http://127.0.97.1:44697/ using document root <none> and password file <none>
I20260928 05:11:43.029002 9236 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:43.029136 9236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:43.035723 9236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:11:43.038968 9270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.040557 9236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:11:43.040711 9236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "8bfc91fb0f954d89bb05f012de5fa481"
format_stamp: "Formatted at 2026-09-28 05:11:40 on dist-test-slave-6718"
server_key: "c06dfba68263752bd5128fdaf4aedf84"
server_key_iv: "575f337f87e14f9ae8b1c399a69354a7"
server_key_version: "encryptionkey@0"
I20260928 05:11:43.041947 9236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:43.064580 9236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:43.069402 9236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:43.069660 9236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:43.070782 9236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:43.072443 9236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:43.072562 9236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.072669 9236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:43.072737 9236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.106504 9236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34739
I20260928 05:11:43.106639 9382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34739 every 8 connection(s)
I20260928 05:11:43.107968 9236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:11:43.109700 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9236
I20260928 05:11:43.110265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37721
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41229
--webserver_port=39773
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:11:43.129637 9385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:43.111280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51173 (local address 127.0.97.62:41229)
0928 05:11:43.112139 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:43.112161 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:43.112956 (+ 795us) server_negotiation.cc:407] Connection header received
0928 05:11:43.115110 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:43.115131 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:43.115573 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:43.115966 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:43.117946 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.120027 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.121796 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.122792 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.122985 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.128761 (+ 5776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51173: BlockingRecv error: recv got EOF from 127.0.97.1:51173 (error 108)
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":272,"threads_started":1}
I20260928 05:11:43.129776 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:43.111332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:51173)
0928 05:11:43.112743 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:43.112793 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:43.114460 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:43.116091 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:43.116099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:43.116953 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:43.117739 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.117768 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.120186 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.120190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.121617 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.121630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.123016 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.123023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.123306 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.123981 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:43.124003 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:43.128464 (+ 4461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":975,"threads_started":1}
W20260928 05:11:43.130580 9383 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41229 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41229: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:43.138804 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:43.131165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:58107)
0928 05:11:43.131328 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:43.131345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:43.131463 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:43.131773 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:43.131777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:43.132100 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:43.132223 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.132230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.133232 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.133238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.134110 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.134125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.134845 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.134850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.135023 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.135650 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:43.135666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:43.138497 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:11:43.138913 9385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:43.131345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58107 (local address 127.0.97.62:41229)
0928 05:11:43.131517 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:43.131524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:43.131541 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:11:43.131588 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:43.131593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:43.131654 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:43.131751 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:43.132358 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.133094 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.134258 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.134710 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.134843 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.138694 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58107: BlockingRecv error: recv got EOF from 127.0.97.1:58107 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:43.147627 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:43.139443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:58095)
0928 05:11:43.139615 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:43.139633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:43.139739 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:43.140041 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:43.140045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:43.140394 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:43.140565 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.140574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.141474 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.141479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.142946 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.142960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.143584 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.143588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.143765 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.144293 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:43.144316 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:43.147369 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:11:43.148093 9385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:43.139490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58095 (local address 127.0.97.62:41229)
0928 05:11:43.139629 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:43.139633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:43.139705 (+ 72us) server_negotiation.cc:407] Connection header received
0928 05:11:43.139861 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:43.139865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:43.139925 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:43.139995 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:43.140685 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.141345 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.143072 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.143450 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.143546 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.147520 (+ 3974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58095: BlockingRecv error: recv got EOF from 127.0.97.1:58095 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 05:11:43.148072 9383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:11:43.430094 9387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:43.430406 9387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:43.430629 9387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:43.437552 9387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:43.437686 9387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:43.437867 9387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:11:43.447264 9387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39773
--tserver_master_addrs=127.0.97.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:43.448585 9387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:43.449787 9387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:43.462347 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:43.462316 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:43.462262 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:43.463119 9387 server_base.cc:1034] running on GCE node
I20260928 05:11:43.463663 9387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:43.464501 9387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:43.465687 9387 hybrid_clock.cc:648] HybridClock initialized: now 1790572303465630 us; error 48 us; skew 500 ppm
I20260928 05:11:43.465968 9387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572303, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:43.468936 9387 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 05:11:43.470216 9387 webserver.cc:460] Webserver started at http://127.0.97.2:39773/ using document root <none> and password file <none>
I20260928 05:11:43.470922 9387 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:43.471062 9387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:43.476027 9387 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:11:43.478669 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.479905 9387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:11:43.480074 9387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "b8b64df4d8da4df4af3e0d7ec80a358a"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "91d0dbb8f12e9cc5036b39b5f213f357"
server_key_iv: "4a567beef6c994149883b6d592147c33"
server_key_version: "encryptionkey@0"
I20260928 05:11:43.480966 9387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:43.504782 9387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:43.508807 9387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:43.509076 9387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:43.510033 9387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:43.511332 9387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:43.511435 9387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.511583 9387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:43.511679 9387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.544018 9387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37721
I20260928 05:11:43.544124 9514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37721 every 8 connection(s)
I20260928 05:11:43.545398 9387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:11:43.552556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9387
I20260928 05:11:43.553284 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44321
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41229
--webserver_port=38219
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:11:43.561177 9385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:43.548005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35039 (local address 127.0.97.62:41229)
0928 05:11:43.548195 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:43.548201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:43.549312 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 05:11:43.551147 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:43.551151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:43.551219 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:43.551311 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:43.552697 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.553820 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.555364 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.556060 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.556210 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.560871 (+ 4661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35039: BlockingRecv error: recv got EOF from 127.0.97.2:35039 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:11:43.561645 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:43.548500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:35039)
0928 05:11:43.549156 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:43.549196 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:43.550897 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:43.551498 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:43.551506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:43.551994 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:43.552515 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.552531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.553969 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.553973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.555224 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.555235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.556259 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.556263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.556443 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.557075 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:43.557095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:43.560624 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
W20260928 05:11:43.562352 9515 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41229 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41229: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:43.571254 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:43.562961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:58809)
0928 05:11:43.563112 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:43.563132 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:43.563224 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:43.563522 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:43.563525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:43.563800 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:43.563922 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.563929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.565530 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.565535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.566430 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.566440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.567465 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.567469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.567587 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.568183 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:43.568200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:43.571065 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:11:43.571647 9385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:43.563048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58809 (local address 127.0.97.62:41229)
0928 05:11:43.563222 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:43.563230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:43.563251 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:11:43.563304 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:43.563312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:43.563399 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:43.563500 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:43.564056 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.565218 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.566604 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.567330 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.567480 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.571345 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58809: BlockingRecv error: recv got EOF from 127.0.97.2:58809 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:43.580703 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:43.572008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:37543)
0928 05:11:43.572212 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:43.572233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:43.572338 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:43.572661 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:43.572668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:43.573086 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:43.573301 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.573316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.574240 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.574249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.576001 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:43.576017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.576698 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.576707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:43.576925 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.577474 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:43.577495 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:43.580419 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 05:11:43.580940 9385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:43.572181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37543 (local address 127.0.97.62:41229)
0928 05:11:43.572335 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:43.572339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:43.572353 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:11:43.572476 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:43.572479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:43.572532 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:43.572599 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:43.573443 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.574103 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.576158 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:43.576555 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:43.576646 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:43.580601 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37543: BlockingRecv error: recv got EOF from 127.0.97.2:37543 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 05:11:43.580962 9515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:11:43.873080 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:43.873374 9518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:43.873529 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:43.880654 9518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:43.880802 9518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:43.880982 9518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:11:43.890671 9518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38219
--tserver_master_addrs=127.0.97.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:43.891924 9518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:43.893138 9518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:43.905912 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:43.905956 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:43.906181 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:43.906937 9518 server_base.cc:1034] running on GCE node
I20260928 05:11:43.907477 9518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:43.908358 9518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:43.909566 9518 hybrid_clock.cc:648] HybridClock initialized: now 1790572303909507 us; error 61 us; skew 500 ppm
I20260928 05:11:43.909849 9518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:11:43 dist-test-slave-6718 krb5kdc[8649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572303, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:11:43.912810 9518 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 05:11:43.914230 9518 webserver.cc:460] Webserver started at http://127.0.97.3:38219/ using document root <none> and password file <none>
I20260928 05:11:43.914947 9518 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:43.915089 9518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:43.919997 9518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:11:43.922771 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.924083 9518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:11:43.924295 9518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "cef05e4e37f94a128213ef79aa1a61b4"
format_stamp: "Formatted at 2026-09-28 05:11:41 on dist-test-slave-6718"
server_key: "db5a4e4f031b35c8c8f28ae4022e9e9d"
server_key_iv: "d5396c62827bc68d528000d20a4ebcd8"
server_key_version: "encryptionkey@0"
I20260928 05:11:43.925280 9518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:43.959407 9518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:43.963750 9518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:43.964038 9518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:43.965123 9518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:11:43.966517 9518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:11:43.966663 9518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:43.966799 9518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:11:43.966897 9518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:44.002588 9518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44321
I20260928 05:11:44.002619 9645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44321 every 8 connection(s)
I20260928 05:11:44.004144 9518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:11:44.005849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9518
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:44.023269 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:44.007359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.97.62:41229)
0928 05:11:44.007577 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:44.007586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:44.007610 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:44.007677 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:44.007683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:44.007762 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:44.007889 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:44.009510 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.010297 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.012289 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.012769 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.012917 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.016113 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:44.016122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:44.016137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:44.016178 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:44.019922 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:44.020635 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:44.020650 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:44.020660 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:44.020866 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:11:44.021330 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:11:44.021334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:11:44.021336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:11:44.021743 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:44.022113 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:44.022599 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 05:11:44.022952 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 05:11:44.026580 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:44.007509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44331 (local address 127.0.97.62:41229)
0928 05:11:44.013384 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:44.013393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:44.013414 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:11:44.014631 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:44.014637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:44.014720 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:44.014876 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:44.016513 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.017898 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.019889 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.020427 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.020522 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.026205 (+ 5683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44331: BlockingRecv error: recv got EOF from 127.0.97.3:44331 (error 108)
Metrics: {"server-negotiator.queue_time_us":5751,"spinlock_wait_cycles":24448,"thread_start_us":5657,"threads_started":1}
I20260928 05:11:44.027307 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:44.007408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:44331)
0928 05:11:44.011061 (+ 3653us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:44.011120 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:44.014336 (+ 3216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:44.015062 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:44.015071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:44.015671 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:44.016293 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.016321 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.018096 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.018100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.019728 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.019740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.020655 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.020660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.020864 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.021588 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:44.021612 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:44.025970 (+ 4358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3322,"thread_start_us":3115,"threads_started":1}
W20260928 05:11:44.028071 9646 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41229 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41229: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:44.036191 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:44.028650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:57347)
0928 05:11:44.028971 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:44.028987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:44.029080 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:44.029405 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:44.029428 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:44.029766 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:44.029950 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.029966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.031003 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.031007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.031979 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.031990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.032581 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.032585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.032720 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.033250 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:44.033271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:44.035986 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237}
W20260928 05:11:44.036350 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:44.028820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57347 (local address 127.0.97.62:41229)
0928 05:11:44.028960 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:44.028964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:44.029030 (+ 66us) server_negotiation.cc:407] Connection header received
0928 05:11:44.029188 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:44.029193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:44.029268 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:44.029365 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:44.030071 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.030858 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.032099 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.032462 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.032548 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.036153 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57347: BlockingRecv error: recv got EOF from 127.0.97.3:57347 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:44.044275 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:44.036886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:39747)
0928 05:11:44.037051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:44.037073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:44.037149 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:44.037390 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:44.037394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:44.037673 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:44.037853 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.037864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.039249 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.039255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.040356 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.040367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.040979 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.040983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.041105 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.041572 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:44.041590 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:44.044074 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 05:11:44.044382 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:44.036974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39747 (local address 127.0.97.62:41229)
0928 05:11:44.037161 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:44.037165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:44.037198 (+ 33us) server_negotiation.cc:407] Connection header received
0928 05:11:44.037233 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:44.037236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:44.037283 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:44.037347 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:44.038023 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.039083 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.040474 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.040854 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.040950 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.044219 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39747: BlockingRecv error: recv got EOF from 127.0.97.3:39747 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 05:11:44.044695 9646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:44.156364 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:44.148608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:48793)
0928 05:11:44.149083 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:44.149099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:44.149211 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:44.149505 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:44.149508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:44.149947 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:44.150134 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.150142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.151112 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.151116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.152071 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.152081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.152727 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.152732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.152854 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.153412 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:44.153429 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:44.156093 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
W20260928 05:11:44.156533 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:44.148673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48793 (local address 127.0.97.62:41229)
0928 05:11:44.148890 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:44.148908 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:44.149178 (+ 270us) server_negotiation.cc:407] Connection header received
0928 05:11:44.149304 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:44.149308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:44.149376 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:44.149456 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:44.150250 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.150999 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.152191 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.152579 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.152664 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.156310 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48793: BlockingRecv error: recv got EOF from 127.0.97.1:48793 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:44 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:44.589933 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:44.581561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:39255)
0928 05:11:44.582072 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:44.582095 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:44.582230 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:44.582608 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:44.582613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:44.583180 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:44.583358 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.583366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.584472 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.584475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.585538 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:44.585549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.586425 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.586430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:44.586560 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.587168 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:44.587184 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:44.589721 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
W20260928 05:11:44.590209 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:44.581630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39255 (local address 127.0.97.62:41229)
0928 05:11:44.581868 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:44.581874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:44.582180 (+ 306us) server_negotiation.cc:407] Connection header received
0928 05:11:44.582340 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:44.582345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:44.582432 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:44.582510 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:44.583488 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.584378 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.585805 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:44.586224 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:44.586371 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:44.589936 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39255: BlockingRecv error: recv got EOF from 127.0.97.2:39255 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 05:11:45 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:45 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:45.056953 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:45.045294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:60299)
0928 05:11:45.045880 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:45.045896 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:45.046059 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:45.046490 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:45.046493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:45.047044 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:45.047225 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:45.047233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.048353 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.048356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:45.049302 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:45.049313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.050017 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.050021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:45.050135 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:45.053793 (+ 3658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:45.053810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:45.056675 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":328,"threads_started":1}
W20260928 05:11:45.057157 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:45.045319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60299 (local address 127.0.97.62:41229)
0928 05:11:45.045505 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:45.045512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:45.045988 (+ 476us) server_negotiation.cc:407] Connection header received
0928 05:11:45.046200 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:45.046208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:45.046342 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:45.046437 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:45.047387 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.048230 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.049425 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.049873 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.049969 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:45.056895 (+ 6926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60299: BlockingRecv error: recv got EOF from 127.0.97.3:60299 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 05:11:45 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:45 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:45.165709 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:45.157483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:38343)
0928 05:11:45.158005 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:45.158020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:45.158176 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:45.158516 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:45.158521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:45.158947 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:45.159085 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:45.159093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.160095 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.160101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:45.161002 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:45.161013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.161731 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.161737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:45.161899 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:45.162671 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:45.162688 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:45.165375 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":297,"threads_started":1}
W20260928 05:11:45.165800 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:45.157543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38343 (local address 127.0.97.62:41229)
0928 05:11:45.157824 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:45.157855 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:45.158111 (+ 256us) server_negotiation.cc:407] Connection header received
0928 05:11:45.158312 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:45.158317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:45.158383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:45.158486 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:45.159223 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.159954 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.161138 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.161575 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.161709 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:45.165562 (+ 3853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38343: BlockingRecv error: recv got EOF from 127.0.97.1:38343 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 28 05:11:45 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:45 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:45.605995 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:45.590794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:39303)
0928 05:11:45.591334 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:45.591355 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:45.591515 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:45.591882 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:45.591888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:45.592314 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:45.592453 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:45.592461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.593605 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.593610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:45.594646 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:45.594664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.595373 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.595379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:45.595544 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:45.602863 (+ 7319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:45.602882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:45.605646 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":293,"threads_started":1}
W20260928 05:11:45.606065 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:45.590871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39303 (local address 127.0.97.62:41229)
0928 05:11:45.591136 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:45.591143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:45.591440 (+ 297us) server_negotiation.cc:407] Connection header received
0928 05:11:45.591647 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:45.591656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:45.591724 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:45.591839 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:45.592600 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.593469 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.594801 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:45.595214 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:45.595366 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:45.605849 (+ 10483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39303: BlockingRecv error: recv got EOF from 127.0.97.2:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 05:11:46 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:46 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:46.066408 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:46.057890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:52133)
0928 05:11:46.058399 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:46.058449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:46.058625 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:46.058910 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:46.058914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:46.059289 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:46.059450 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:46.059458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.060473 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.060476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:46.061752 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:46.061766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.062781 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.062786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:46.062891 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:46.063485 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:46.063502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:46.066170 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
W20260928 05:11:46.066591 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:46.058112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52133 (local address 127.0.97.62:41229)
0928 05:11:46.058329 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:46.058335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:46.058545 (+ 210us) server_negotiation.cc:407] Connection header received
0928 05:11:46.058722 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:46.058725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:46.058795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:46.058874 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:46.059575 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.060343 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.061905 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.062641 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.062733 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:46.066348 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52133: BlockingRecv error: recv got EOF from 127.0.97.3:52133 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 05:11:46 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:46 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:46.175057 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:46.166747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:49049)
0928 05:11:46.167258 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:46.167279 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:46.167439 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:46.167801 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:46.167807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:46.168239 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:46.168444 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:46.168457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.169618 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.169626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:46.170685 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:46.170700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.171349 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.171358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:46.171482 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:46.172002 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:46.172018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:46.174701 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":237,"threads_started":1}
W20260928 05:11:46.175122 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:46.166942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49049 (local address 127.0.97.62:41229)
0928 05:11:46.167151 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:46.167157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:46.167366 (+ 209us) server_negotiation.cc:407] Connection header received
0928 05:11:46.167576 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:46.167582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:46.167665 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:46.167740 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:46.168615 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.169486 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.170820 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.171231 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.171324 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:46.174902 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49049: BlockingRecv error: recv got EOF from 127.0.97.1:49049 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 05:11:46 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:46 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:46.615046 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:46.607127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:38353)
0928 05:11:46.607655 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:46.607684 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:46.607794 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:46.608089 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:46.608092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:46.608459 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:46.608615 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:46.608624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.609603 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.609607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:46.610521 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:46.610572 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.611239 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.611243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:46.611403 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:46.611987 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:46.612005 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:46.614803 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":301,"threads_started":1}
W20260928 05:11:46.615208 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:46.607200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38353 (local address 127.0.97.62:41229)
0928 05:11:46.607412 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:46.607421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:46.607763 (+ 342us) server_negotiation.cc:407] Connection header received
0928 05:11:46.607893 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:46.607897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:46.607957 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:46.608038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:46.608741 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.609464 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.610668 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:46.611078 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:46.611191 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:46.614944 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38353: BlockingRecv error: recv got EOF from 127.0.97.2:38353 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 05:11:47 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:47 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:47.075619 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:47.067446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:43989)
0928 05:11:47.067879 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:47.067895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:47.068020 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:47.068392 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:47.068395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:47.068863 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:47.069094 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:47.069111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.070273 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.070277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:47.071158 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:47.071169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.071855 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.071859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:47.071996 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:47.072702 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:47.072719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:47.075366 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
W20260928 05:11:47.075800 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:47.067521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43989 (local address 127.0.97.62:41229)
0928 05:11:47.067732 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:47.067740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:47.067989 (+ 249us) server_negotiation.cc:407] Connection header received
0928 05:11:47.068122 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:47.068129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:47.068206 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:47.068375 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:47.069252 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.070158 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.071288 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.071731 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.071830 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:47.075582 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43989: BlockingRecv error: recv got EOF from 127.0.97.3:43989 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 05:11:47 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:47 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:47.184206 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:47.176127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:38695)
0928 05:11:47.176673 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:47.176689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:47.176841 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:47.177169 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:47.177173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:47.177543 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:47.177678 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:47.177685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.178741 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.178748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:47.179687 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:47.179698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.180401 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.180406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:47.180587 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:47.181204 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:47.181222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:47.183963 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
W20260928 05:11:47.184648 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:47.176227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38695 (local address 127.0.97.62:41229)
0928 05:11:47.176424 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:47.176433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:47.176782 (+ 349us) server_negotiation.cc:407] Connection header received
0928 05:11:47.176970 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:47.176975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:47.177049 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:47.177146 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:47.177818 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.178616 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.179827 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.180261 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.180390 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:47.184239 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38695: BlockingRecv error: recv got EOF from 127.0.97.1:38695 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 05:11:47 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:47 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:47.623999 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:47.616261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:43841)
0928 05:11:47.616757 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:47.616782 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:47.616919 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:47.617187 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:47.617190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:47.617543 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:47.617689 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:47.617697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.618684 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.618688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:47.619651 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:47.619662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.620463 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.620467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:47.620586 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:47.621118 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:47.621134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:47.623784 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":418,"threads_started":1}
W20260928 05:11:47.624261 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:47.616343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43841 (local address 127.0.97.62:41229)
0928 05:11:47.616522 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:47.616530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:47.616866 (+ 336us) server_negotiation.cc:407] Connection header received
0928 05:11:47.617004 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:47.617007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:47.617068 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:47.617148 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:47.617806 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.618572 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.619768 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:47.620339 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:47.620436 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:47.624000 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43841: BlockingRecv error: recv got EOF from 127.0.97.2:43841 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 05:11:48 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:48 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:48.084967 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:48.076679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:50171)
0928 05:11:48.077155 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:48.077170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:48.077277 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:48.077603 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:48.077606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:48.077933 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:48.078076 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:48.078084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.079216 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.079219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:48.080233 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:48.080246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.081158 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.081164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:48.081352 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:48.081921 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:48.081937 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:48.084712 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
W20260928 05:11:48.085266 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:48.076719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50171 (local address 127.0.97.62:41229)
0928 05:11:48.076929 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:48.076939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:48.077251 (+ 312us) server_negotiation.cc:407] Connection header received
0928 05:11:48.077404 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:48.077407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:48.077477 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:48.077558 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:48.078232 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.079095 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.080417 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.080996 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.081149 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:48.084955 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50171: BlockingRecv error: recv got EOF from 127.0.97.3:50171 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 05:11:48 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:48 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:48.193573 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:48.185157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:38127)
0928 05:11:48.185723 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:48.185738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:48.185886 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:48.186228 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:48.186233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:48.186745 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:48.186904 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:48.186913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.187909 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.187917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:48.188823 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:48.188835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.189508 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.189513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:48.189679 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:48.190267 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:48.190283 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:48.193157 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
W20260928 05:11:48.193572 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:48.185343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38127 (local address 127.0.97.62:41229)
0928 05:11:48.185559 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:48.185566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:48.185833 (+ 267us) server_negotiation.cc:407] Connection header received
0928 05:11:48.186020 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:48.186025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:48.186099 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:48.186200 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:48.187051 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.187761 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.188963 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.189368 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.189499 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:48.193359 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38127: BlockingRecv error: recv got EOF from 127.0.97.1:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 05:11:48 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:48 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:48.634163 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:48.624946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:50411)
0928 05:11:48.625535 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:48.625584 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:48.625727 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:48.626119 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:48.626125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:48.626733 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:48.626945 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:48.626954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.628419 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.628423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:48.629421 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:48.629440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.630445 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.630449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:48.630627 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:48.631267 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:48.631284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:48.633936 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":284,"threads_started":1}
W20260928 05:11:48.634305 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:48.625092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50411 (local address 127.0.97.62:41229)
0928 05:11:48.625292 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:48.625317 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:48.625685 (+ 368us) server_negotiation.cc:407] Connection header received
0928 05:11:48.625895 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:48.625900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:48.625975 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:48.626072 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:48.627094 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.628265 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.629559 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:48.630278 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:48.630392 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:48.634115 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50411: BlockingRecv error: recv got EOF from 127.0.97.2:50411 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 05:11:49 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:49 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:49.094743 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:49.085933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:60979)
0928 05:11:49.086462 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:49.086500 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:49.086671 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:49.087085 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:49.087090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:49.087644 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:49.087887 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:49.087902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.089097 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.089101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:49.090050 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:49.090062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.090773 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.090777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:49.090915 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:49.091495 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:49.091516 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:49.094409 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":271,"threads_started":1}
W20260928 05:11:49.094921 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:49.085978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60979 (local address 127.0.97.62:41229)
0928 05:11:49.086141 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:49.086145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:49.086615 (+ 470us) server_negotiation.cc:407] Connection header received
0928 05:11:49.086813 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:49.086818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:49.086897 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:49.087002 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:49.088078 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.088953 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.090174 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.090601 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.090701 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:49.094683 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60979: BlockingRecv error: recv got EOF from 127.0.97.3:60979 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 05:11:49 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:49 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:49.202677 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:49.194562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:46115)
0928 05:11:49.195092 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:49.195114 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:49.195293 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:49.195647 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:49.195654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:49.196056 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:49.196193 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:49.196201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.197195 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.197201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:49.198080 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:49.198091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.198936 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.198943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:49.199101 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:49.199649 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:49.199665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:49.202414 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":454,"threads_started":1}
W20260928 05:11:49.202852 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:49.194652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46115 (local address 127.0.97.62:41229)
0928 05:11:49.194896 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:49.194902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:49.195196 (+ 294us) server_negotiation.cc:407] Connection header received
0928 05:11:49.195428 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:49.195435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:49.195505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:49.195613 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:49.196337 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.197047 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.198224 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.198758 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.198906 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:49.202625 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46115: BlockingRecv error: recv got EOF from 127.0.97.1:46115 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 05:11:49 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:49 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:49.643072 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:49.635158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:39711)
0928 05:11:49.635654 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:49.635680 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:49.635810 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:49.636096 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:49.636099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:49.636463 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:49.636616 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:49.636624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.637568 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.637572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:49.638476 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:49.638488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.639208 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.639212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:49.639336 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:49.639882 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:49.639898 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:49.642751 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"spinlock_wait_cycles":27520,"thread_start_us":363,"threads_started":1}
W20260928 05:11:49.643175 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:49.635209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39711 (local address 127.0.97.62:41229)
0928 05:11:49.635402 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:49.635415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:49.635752 (+ 337us) server_negotiation.cc:407] Connection header received
0928 05:11:49.635911 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:49.635915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:49.635976 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:49.636053 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:49.636737 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.637426 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.638661 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:49.639078 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:49.639169 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:49.642942 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39711: BlockingRecv error: recv got EOF from 127.0.97.2:39711 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 05:11:50 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:50 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:50.109889 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:50.095732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:46295)
0928 05:11:50.096235 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:50.096253 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:50.096387 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:50.096710 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:50.096714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:50.097095 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:50.097241 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:50.097249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.098275 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.098284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:50.099285 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:50.099296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.099967 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.099972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:50.100110 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:50.105519 (+ 5409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:50.105538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:50.109498 (+ 3960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":296,"threads_started":1}
W20260928 05:11:50.110567 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:50.095822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46295 (local address 127.0.97.62:41229)
0928 05:11:50.096096 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:50.096102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:50.096337 (+ 235us) server_negotiation.cc:407] Connection header received
0928 05:11:50.096490 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:50.096493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:50.096554 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:50.096632 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:50.097363 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.098126 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.099405 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.099787 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.099880 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:50.109832 (+ 9952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46295: BlockingRecv error: recv got EOF from 127.0.97.3:46295 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 28 05:11:50 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:50 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:50.211618 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:50.203531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:49609)
0928 05:11:50.204061 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:50.204077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:50.204200 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:50.204499 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:50.204503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:50.204867 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:50.205044 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:50.205052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.205970 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.205974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:50.206972 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:50.206982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.207663 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.207667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:50.207768 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:50.208372 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:50.208389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:50.211262 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":242,"threads_started":1}
W20260928 05:11:50.211685 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:50.203596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49609 (local address 127.0.97.62:41229)
0928 05:11:50.203813 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:50.203822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:50.204162 (+ 340us) server_negotiation.cc:407] Connection header received
0928 05:11:50.204299 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:50.204302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:50.204361 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:50.204441 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:50.205157 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.205854 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.207091 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.207539 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.207631 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:50.211468 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49609: BlockingRecv error: recv got EOF from 127.0.97.1:49609 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 05:11:50 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:50 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:50.652642 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:50.644041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:51933)
0928 05:11:50.644666 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:50.644686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:50.644800 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:50.645088 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:50.645091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:50.645455 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:50.645629 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:50.645637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.646676 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.646680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:50.647612 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:50.647624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.648410 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.648414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:50.648529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:50.649088 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:50.649104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:50.652293 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":273,"threads_started":1}
W20260928 05:11:50.652762 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:50.644093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51933 (local address 127.0.97.62:41229)
0928 05:11:50.644271 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:50.644276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:50.644757 (+ 481us) server_negotiation.cc:407] Connection header received
0928 05:11:50.644902 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:50.644905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:50.644966 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:50.645048 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:50.645751 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.646472 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.647763 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:50.648282 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:50.648384 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:50.652521 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51933: BlockingRecv error: recv got EOF from 127.0.97.2:51933 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 05:11:51 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:51 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:51.119241 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:51.110941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:42003)
0928 05:11:51.111515 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:51.111531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:51.111640 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:51.111938 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:51.111942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:51.112333 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:51.112485 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:51.112493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.113524 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.113532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:51.114462 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:51.114475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.115290 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.115296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:51.115514 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:51.116126 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:51.116144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:51.119020 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":264,"threads_started":1}
W20260928 05:11:51.119477 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:51.111106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42003 (local address 127.0.97.62:41229)
0928 05:11:51.111282 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:51.111286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:51.111626 (+ 340us) server_negotiation.cc:407] Connection header received
0928 05:11:51.111738 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:51.111742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:51.111803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:51.111894 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:51.112618 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.113399 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.114659 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.115124 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.115275 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:51.119245 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42003: BlockingRecv error: recv got EOF from 127.0.97.3:42003 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 05:11:51 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:51 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:51.220431 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:51.212694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:54763)
0928 05:11:51.213229 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:51.213245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:51.213352 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:51.213624 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:51.213627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:51.214019 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:51.214176 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:51.214184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.215254 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.215257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:51.216288 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:51.216299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.216945 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.216949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:51.217071 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:51.217615 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:51.217631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:51.220207 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":312,"threads_started":1}
W20260928 05:11:51.220589 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:51.212831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54763 (local address 127.0.97.62:41229)
0928 05:11:51.213001 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:51.213005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:51.213316 (+ 311us) server_negotiation.cc:407] Connection header received
0928 05:11:51.213443 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:51.213446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:51.213504 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:51.213584 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:51.214292 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.215134 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.216402 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.216802 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.216892 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:51.220380 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54763: BlockingRecv error: recv got EOF from 127.0.97.1:54763 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 05:11:51 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:51 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:51.661693 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:51.653619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:54677)
0928 05:11:51.654088 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:51.654107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:51.654263 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:51.654630 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:51.654635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:51.655037 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:51.655180 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:51.655188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.656367 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.656373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:51.657283 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:51.657294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.658002 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.658008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:51.658174 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:51.658802 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:51.658819 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:51.661338 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":245,"threads_started":1}
W20260928 05:11:51.661823 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:51.653704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54677 (local address 127.0.97.62:41229)
0928 05:11:51.653958 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:51.653964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:51.654195 (+ 231us) server_negotiation.cc:407] Connection header received
0928 05:11:51.654384 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:51.654389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:51.654470 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:51.654606 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:51.655324 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.656226 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.657423 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:51.657825 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:51.657948 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:51.661579 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54677: BlockingRecv error: recv got EOF from 127.0.97.2:54677 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 05:11:52 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:52 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:52.128376 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:52.120200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:48141)
0928 05:11:52.120695 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:52.120711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:52.120880 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:52.121224 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:52.121228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:52.121645 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:52.121788 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:52.121796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.122849 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.122854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:52.123901 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:52.123913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.124639 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.124645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:52.124814 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:52.125360 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:52.125376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:52.128155 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
W20260928 05:11:52.128594 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:52.120278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48141 (local address 127.0.97.62:41229)
0928 05:11:52.120505 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:52.120514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:52.120810 (+ 296us) server_negotiation.cc:407] Connection header received
0928 05:11:52.121023 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:52.121028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:52.121100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:52.121197 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:52.121930 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.122719 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.124050 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.124457 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.124583 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:52.128348 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48141: BlockingRecv error: recv got EOF from 127.0.97.3:48141 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 05:11:52 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:52 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:52.229458 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:52.221341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:55619)
0928 05:11:52.221864 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:52.221880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:52.222037 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:52.222364 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:52.222369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:52.222784 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:52.222953 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:52.222962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.224040 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.224047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:52.224970 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:52.224981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.225664 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.225669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:52.225828 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:52.226433 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:52.226450 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:52.229223 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":260,"threads_started":1}
W20260928 05:11:52.229662 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:52.221427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55619 (local address 127.0.97.62:41229)
0928 05:11:52.221665 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:52.221672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:52.221968 (+ 296us) server_negotiation.cc:407] Connection header received
0928 05:11:52.222160 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:52.222167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:52.222231 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:52.222332 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:52.223109 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.223867 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.225111 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.225511 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.225635 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:52.229411 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55619: BlockingRecv error: recv got EOF from 127.0.97.1:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 05:11:52 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:52 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:52.670399 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:52.662574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:51121)
0928 05:11:52.663040 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:52.663059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:52.663167 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:52.663443 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:52.663446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:52.663817 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:52.663972 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:52.663980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.664953 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.664957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:52.665883 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:52.665893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.666644 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.666649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:52.666761 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:52.667394 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:52.667412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:52.670168 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
W20260928 05:11:52.670603 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:52.662772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51121 (local address 127.0.97.62:41229)
0928 05:11:52.662990 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:52.662996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:52.663131 (+ 135us) server_negotiation.cc:407] Connection header received
0928 05:11:52.663261 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:52.663265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:52.663325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:52.663413 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:52.664092 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.664825 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.666012 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:52.666446 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:52.666606 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:52.670349 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51121: BlockingRecv error: recv got EOF from 127.0.97.2:51121 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 05:11:53 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:53 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:53.137292 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:53.129294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:45405)
0928 05:11:53.129803 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:53.129819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:53.129965 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:53.130301 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:53.130306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:53.130738 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:53.130902 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:53.130910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.131869 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.131874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:53.132812 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:53.132823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.133510 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.133516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:53.133691 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:53.134239 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:53.134255 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:53.136943 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":296,"threads_started":1}
W20260928 05:11:53.137357 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:53.129311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45405 (local address 127.0.97.62:41229)
0928 05:11:53.129526 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:53.129534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:53.129905 (+ 371us) server_negotiation.cc:407] Connection header received
0928 05:11:53.130097 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:53.130102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:53.130169 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:53.130277 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:53.131058 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.131742 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.132950 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.133356 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.133490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:53.137143 (+ 3653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45405: BlockingRecv error: recv got EOF from 127.0.97.3:45405 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 05:11:53 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:53 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:53.238844 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:53.230401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:58885)
0928 05:11:53.230976 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:53.230998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:53.231126 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:53.231474 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:53.231477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:53.231814 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:53.231956 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:53.231965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.232918 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.232921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:53.233935 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:53.233948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.235001 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.235009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:53.235189 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:53.235844 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:53.235861 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:53.238626 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
W20260928 05:11:53.239003 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:53.230485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58885 (local address 127.0.97.62:41229)
0928 05:11:53.230694 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:53.230703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:53.231082 (+ 379us) server_negotiation.cc:407] Connection header received
0928 05:11:53.231258 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:53.231262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:53.231319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:53.231397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:53.232074 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.232792 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.234115 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.234809 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.234900 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:53.238749 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58885: BlockingRecv error: recv got EOF from 127.0.97.1:58885 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 05:11:53 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:53 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:53.679512 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:53.671412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:53473)
0928 05:11:53.671881 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:53.671901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:53.672031 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:53.672318 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:53.672321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:53.672694 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:53.672827 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:53.672835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.673779 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.673783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:53.674735 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:53.674746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.675634 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.675640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:53.675816 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:53.676438 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:53.676454 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:53.679275 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":274,"threads_started":1}
W20260928 05:11:53.679709 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:53.671477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53473 (local address 127.0.97.62:41229)
0928 05:11:53.671714 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:53.671720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:53.672000 (+ 280us) server_negotiation.cc:407] Connection header received
0928 05:11:53.672129 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:53.672132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:53.672191 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:53.672269 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:53.672955 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.673642 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.674868 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:53.675454 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:53.675627 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:53.679461 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53473: BlockingRecv error: recv got EOF from 127.0.97.2:53473 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 05:11:54 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:54 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:54.146099 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:54.138138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:34889)
0928 05:11:54.138705 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:54.138720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:54.138864 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:54.139202 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:54.139207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:54.139594 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:54.139726 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:54.139734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.140722 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.140728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:54.141638 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:54.141648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.142328 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.142333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:54.142513 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:54.143123 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:54.143140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:54.145747 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":287,"threads_started":1}
W20260928 05:11:54.146247 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:54.138310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34889 (local address 127.0.97.62:41229)
0928 05:11:54.138617 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:54.138622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:54.138797 (+ 175us) server_negotiation.cc:407] Connection header received
0928 05:11:54.138982 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:54.138993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:54.139077 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:54.139180 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:54.139867 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.140590 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.141781 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.142177 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.142304 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:54.145930 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34889: BlockingRecv error: recv got EOF from 127.0.97.3:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 28 05:11:54 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:54 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:54.248008 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:54.239802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:35077)
0928 05:11:54.240253 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:54.240269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:54.240376 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:54.240702 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:54.240705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:54.241085 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:54.241250 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:54.241258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.242487 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.242491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:54.243591 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:54.243602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.244253 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.244257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:54.244365 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:54.244921 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:54.244937 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:54.247775 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
W20260928 05:11:54.248186 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:54.239859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35077 (local address 127.0.97.62:41229)
0928 05:11:54.240061 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:54.240069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:54.240349 (+ 280us) server_negotiation.cc:407] Connection header received
0928 05:11:54.240478 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:54.240482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:54.240579 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:54.240660 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:54.241365 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.242363 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.243705 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.244131 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.244224 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:54.247988 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35077: BlockingRecv error: recv got EOF from 127.0.97.1:35077 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 05:11:54 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:54 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:54.688985 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:54.680380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:43175)
0928 05:11:54.680875 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:54.680898 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:54.681024 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:54.681407 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:54.681411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:54.681959 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:54.682220 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:54.682231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.683551 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.683555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:54.684648 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:54.684660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.685389 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.685393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:54.685530 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:54.686066 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:54.686083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:54.688741 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
W20260928 05:11:54.689124 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:54.680493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43175 (local address 127.0.97.62:41229)
0928 05:11:54.680671 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:54.680675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:54.680977 (+ 302us) server_negotiation.cc:407] Connection header received
0928 05:11:54.681125 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:54.681135 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:54.681215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:54.681334 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:54.682367 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.683420 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.684777 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:54.685261 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:54.685350 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:54.688911 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43175: BlockingRecv error: recv got EOF from 127.0.97.2:43175 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 05:11:55 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:55 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:55.159512 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:55.147142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:58047)
0928 05:11:55.147623 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:55.147639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:55.147745 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:55.148017 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:55.148021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:55.148390 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:55.148579 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:55.148587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.149539 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.149542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:55.150389 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:55.150398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.151031 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.151035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:55.151141 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:55.156308 (+ 5167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:55.156324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:55.159284 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":243,"threads_started":1}
W20260928 05:11:55.159695 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:55.147213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58047 (local address 127.0.97.62:41229)
0928 05:11:55.147468 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:55.147474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:55.147713 (+ 239us) server_negotiation.cc:407] Connection header received
0928 05:11:55.147834 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:55.147838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:55.147899 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:55.147978 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:55.148712 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.149390 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.150506 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.150914 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.151002 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:55.159488 (+ 8486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58047: BlockingRecv error: recv got EOF from 127.0.97.3:58047 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 05:11:55 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:55 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:55.257409 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:55.248961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:39001)
0928 05:11:55.249552 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:55.249568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:55.249730 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:55.250076 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:55.250081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:55.250479 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:55.250641 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:55.250649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.251663 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.251672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:55.252790 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:55.252801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.253658 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.253662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:55.253796 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:55.254360 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:55.254377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:55.257168 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":347,"threads_started":1}
W20260928 05:11:55.257694 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:55.249002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39001 (local address 127.0.97.62:41229)
0928 05:11:55.249325 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:55.249333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:55.249665 (+ 332us) server_negotiation.cc:407] Connection header received
0928 05:11:55.249873 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:55.249882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:55.249942 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:55.250045 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:55.250790 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.251530 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.252921 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.253491 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.253635 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:55.257383 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39001: BlockingRecv error: recv got EOF from 127.0.97.1:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 28 05:11:55 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:55 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:55.700590 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:55.690206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:40045)
0928 05:11:55.690813 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:55.690833 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:55.690933 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:55.691322 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:55.691328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:55.691790 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:55.691977 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:55.691988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.693669 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.693674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:55.695372 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:55.695386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.696448 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.696453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:55.696620 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:55.697343 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:55.697362 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:55.700188 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":374,"threads_started":1}
W20260928 05:11:55.700687 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:55.690712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40045 (local address 127.0.97.62:41229)
0928 05:11:55.690875 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:55.690879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:55.690895 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:11:55.691059 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:55.691067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:55.691169 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:55.691283 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:55.692141 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.693198 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.695532 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:55.696141 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:55.696228 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:55.700405 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40045: BlockingRecv error: recv got EOF from 127.0.97.2:40045 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 05:11:56 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:56 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:56.168406 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:56.160386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:55467)
0928 05:11:56.160894 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:56.160910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:56.161073 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:56.161399 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:56.161403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:56.161806 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:56.161948 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:56.161955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.163051 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.163058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:56.163975 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:56.163986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.164675 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.164682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:56.164852 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:56.165432 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:56.165450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:56.168187 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":295,"threads_started":1}
W20260928 05:11:56.168637 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:56.160463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55467 (local address 127.0.97.62:41229)
0928 05:11:56.160704 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:56.160713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:56.161001 (+ 288us) server_negotiation.cc:407] Connection header received
0928 05:11:56.161194 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:56.161200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:56.161269 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:56.161370 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:56.162102 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.162918 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.164114 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.164518 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.164643 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:56.168357 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55467: BlockingRecv error: recv got EOF from 127.0.97.3:55467 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 05:11:56 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:56 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:56.266775 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:56.258370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:40547)
0928 05:11:56.259002 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:56.259018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:56.259175 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:56.259496 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:56.259500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:56.259911 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:56.260090 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:56.260098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.261099 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.261105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:56.262034 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:56.262044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.262783 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.262789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:56.262943 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:56.263610 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:56.263636 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:56.266474 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":366,"threads_started":1}
W20260928 05:11:56.266981 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:56.258500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40547 (local address 127.0.97.62:41229)
0928 05:11:56.258836 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:56.258843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:56.259111 (+ 268us) server_negotiation.cc:407] Connection header received
0928 05:11:56.259295 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:56.259302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:56.259365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:56.259468 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:56.260232 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.260956 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.262174 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.262617 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.262748 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:56.266754 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40547: BlockingRecv error: recv got EOF from 127.0.97.1:40547 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 28 05:11:56 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:56 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:56.709662 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:56.701645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:50019)
0928 05:11:56.702285 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:56.702309 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:56.702455 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:56.702889 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:56.702894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:56.703255 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:56.703421 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:56.703428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.704381 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.704385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:56.705298 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:56.705309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.706022 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.706027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:56.706138 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:56.706703 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:56.706719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:56.709336 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":384,"threads_started":1}
W20260928 05:11:56.709754 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:56.701838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50019 (local address 127.0.97.62:41229)
0928 05:11:56.702023 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:56.702028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:56.702399 (+ 371us) server_negotiation.cc:407] Connection header received
0928 05:11:56.702627 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:56.702631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:56.702736 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:56.702842 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:56.703550 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.704267 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.705437 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:56.705879 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:56.705971 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:56.709525 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50019: BlockingRecv error: recv got EOF from 127.0.97.2:50019 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 05:11:57 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:57 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:57.177440 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:57.169389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:47269)
0928 05:11:57.169900 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:57.169920 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:57.170074 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:57.170373 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:57.170376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:57.170767 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:57.170943 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:57.170951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.171980 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.171983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:57.172883 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:57.172893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.173600 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.173604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:57.173714 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:57.174248 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:57.174264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:57.177106 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":302,"threads_started":1}
W20260928 05:11:57.177518 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:57.169490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47269 (local address 127.0.97.62:41229)
0928 05:11:57.169691 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:57.169697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:57.170010 (+ 313us) server_negotiation.cc:407] Connection header received
0928 05:11:57.170187 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:57.170191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:57.170253 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:57.170335 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:57.171096 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.171863 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.173019 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.173408 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.173538 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:57.177293 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47269: BlockingRecv error: recv got EOF from 127.0.97.3:47269 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 05:11:57 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:57 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:57.276424 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:57.267734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:54835)
0928 05:11:57.268225 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:57.268240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:57.268393 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:57.268760 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:57.268764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:57.269178 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:57.269318 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:57.269325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.270295 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.270301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:57.271277 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:57.271288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.272017 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.272023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:57.272195 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:57.272790 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:57.272807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:57.276076 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
W20260928 05:11:57.276490 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:57.267796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54835 (local address 127.0.97.62:41229)
0928 05:11:57.268087 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:57.268094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:57.268321 (+ 227us) server_negotiation.cc:407] Connection header received
0928 05:11:57.268513 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:57.268520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:57.268625 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:57.268730 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:57.269454 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.270155 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.271414 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.271874 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.272009 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:57.276250 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54835: BlockingRecv error: recv got EOF from 127.0.97.1:54835 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 28 05:11:57 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:57 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:57.719135 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:57.710761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:34799)
0928 05:11:57.711301 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:57.711322 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:57.711530 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:57.711887 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:57.711892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:57.712277 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:57.712433 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:57.712443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.713563 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.713570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:57.714780 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:57.714792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.715489 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.715493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:57.715606 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:57.716202 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:57.716252 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:57.718877 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
W20260928 05:11:57.719228 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:57.710920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34799 (local address 127.0.97.62:41229)
0928 05:11:57.711167 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:57.711174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:57.711416 (+ 242us) server_negotiation.cc:407] Connection header received
0928 05:11:57.711659 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:57.711665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:57.711747 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:57.711855 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:57.712578 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.713412 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.714946 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:57.715347 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:57.715452 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:57.719051 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34799: BlockingRecv error: recv got EOF from 127.0.97.2:34799 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 05:11:58 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:58 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:58.186412 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:58.178422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.3:41511)
0928 05:11:58.178921 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:58.178936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:58.179040 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:58.179347 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:58.179351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:58.179692 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:58.179831 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:58.179838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.180852 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.180855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:58.181742 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:58.181752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.182701 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.182705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:58.182820 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:58.183360 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:58.183377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:58.186192 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":316,"threads_started":1}
W20260928 05:11:58.186693 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:58.178475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41511 (local address 127.0.97.62:41229)
0928 05:11:58.178656 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:58.178660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:58.179016 (+ 356us) server_negotiation.cc:407] Connection header received
0928 05:11:58.179147 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:58.179150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:58.179227 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:58.179309 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:58.179966 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.180724 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.181870 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.182486 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.182699 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:58.186386 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41511: BlockingRecv error: recv got EOF from 127.0.97.3:41511 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 05:11:58 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:58 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:58.286226 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:58.277461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.1:45299)
0928 05:11:58.277923 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:58.277938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:58.278081 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:58.278404 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:58.278408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:58.278954 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:58.279129 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:58.279138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.280110 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.280113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:58.281050 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:58.281061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.281709 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.281713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:58.281824 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:58.282506 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:58.282558 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:58.285871 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":256,"threads_started":1}
W20260928 05:11:58.286311 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:58.277612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45299 (local address 127.0.97.62:41229)
0928 05:11:58.277817 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:58.277823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:58.278013 (+ 190us) server_negotiation.cc:407] Connection header received
0928 05:11:58.278175 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:58.278178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:58.278251 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:58.278331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:58.279268 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.279977 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.281171 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.281580 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.281671 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:58.286086 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45299: BlockingRecv error: recv got EOF from 127.0.97.1:45299 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 05:11:58 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 05:11:58 dist-test-slave-6718 krb5kdc[8649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 05:11:58.728579 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:58.720292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41229 (local address 127.0.97.2:42159)
0928 05:11:58.720844 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 05:11:58.720863 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:11:58.721003 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:11:58.721342 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:11:58.721347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:11:58.721768 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:11:58.721920 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:58.721929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.722991 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.722997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:58.724045 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:11:58.724062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.724729 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.724735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:11:58.724906 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:58.725515 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:11:58.725531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:11:58.728243 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41229: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":312,"threads_started":1}
W20260928 05:11:58.728626 9649 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:11:58.720335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42159 (local address 127.0.97.62:41229)
0928 05:11:58.720552 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:58.720557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:58.720942 (+ 385us) server_negotiation.cc:407] Connection header received
0928 05:11:58.721129 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:58.721134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:58.721208 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:58.721317 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:58.722057 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.722858 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.724187 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:58.724590 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:58.724721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:58.728386 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42159: BlockingRecv error: recv got EOF from 127.0.97.2:42159 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:11:59.017112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9236
I20260928 05:11:59.029309 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:59.018216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.97.1:34739)
0928 05:11:59.018825 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:59.018832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:59.018856 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:59.018909 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:59.018916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:59.019124 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:59.019279 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:59.020629 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.021641 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.022950 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.023477 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.023596 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:59.026171 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:59.026180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:59.026193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:59.026229 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:59.028729 (+ 2500us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:11:59.029006 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40792: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":342,"threads_started":1}
W20260928 05:11:59.029412 9696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40792: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:11:59.029475 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9236: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34739: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:11:59.029680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9236
I20260928 05:11:59.046572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9387
I20260928 05:11:59.057974 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:59.047453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.0.97.2:37721)
0928 05:11:59.047958 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:59.047966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:59.047994 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:11:59.048049 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:59.048056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:59.048278 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:59.048424 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:59.049660 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.050798 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.052010 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.052524 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.052627 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:59.054951 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:59.054959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:59.054971 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:59.055000 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:59.057467 (+ 2467us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:11:59.057707 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39044: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":227,"threads_started":1}
W20260928 05:11:59.058072 9697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39044: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:11:59.058089 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9387: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:37721: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:11:59.058400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9387
I20260928 05:11:59.074340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9518
I20260928 05:11:59.085153 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:59.075207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.0.97.3:44321)
0928 05:11:59.075729 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:59.075736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:59.075760 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:11:59.075819 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:59.075826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:59.076028 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:59.076181 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:59.077372 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.078316 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.079469 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.079967 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.080066 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:59.082280 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:59.082287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:59.082301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:11:59.082329 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:59.084710 (+ 2381us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 05:11:59.084929 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32858: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
W20260928 05:11:59.085266 9698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32858: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 05:11:59.085495 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9518: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44321: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 05:11:59.085624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9518
I20260928 05:11:59.101704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9167
I20260928 05:11:59.102826 9229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:11:59.214087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9167
2026-09-28T05:11:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20949 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T05:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:11:59Z Disabled control of system clock
I20260928 05:11:59.286520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37365
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 05:11:59.602403 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:11:59.602727 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:11:59.602808 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:11:59.610086 9703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:11:59.610199 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:11:59.610237 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:11:59.610285 9703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:11:59.610329 9703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:11:59.619554 9703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:11:59.620836 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:11:59.622622 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:11:59.634546 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:59.634641 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:11:59.634605 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:11:59.635108 9703 server_base.cc:1034] running on GCE node
I20260928 05:11:59.635931 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:11:59.637194 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:11:59.638423 9703 hybrid_clock.cc:648] HybridClock initialized: now 1790572319638395 us; error 44 us; skew 500 ppm
I20260928 05:11:59.638803 9703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:11:59.641047 9703 webserver.cc:460] Webserver started at http://127.0.97.62:41983/ using document root <none> and password file <none>
I20260928 05:11:59.641737 9703 fs_manager.cc:359] Metadata directory not provided
I20260928 05:11:59.641866 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:11:59.642230 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:11:59.645162 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "0e11c57ecbb44659a62f653cd008f47e"
format_stamp: "Formatted at 2026-09-28 05:11:59 on dist-test-slave-6718"
server_key: "3ec812bfb921393f174adb7c1dadcde2"
server_key_iv: "c5f31299de2a1c34bbfd8dae309b1f9c"
server_key_version: "encryptionkey@0"
I20260928 05:11:59.645970 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e11c57ecbb44659a62f653cd008f47e"
format_stamp: "Formatted at 2026-09-28 05:11:59 on dist-test-slave-6718"
server_key: "3ec812bfb921393f174adb7c1dadcde2"
server_key_iv: "c5f31299de2a1c34bbfd8dae309b1f9c"
server_key_version: "encryptionkey@0"
I20260928 05:11:59.651503 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:11:59.655170 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:11:59.656703 9703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:11:59.656908 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "0e11c57ecbb44659a62f653cd008f47e"
format_stamp: "Formatted at 2026-09-28 05:11:59 on dist-test-slave-6718"
server_key: "3ec812bfb921393f174adb7c1dadcde2"
server_key_iv: "c5f31299de2a1c34bbfd8dae309b1f9c"
server_key_version: "encryptionkey@0"
I20260928 05:11:59.657155 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:11:59.668953 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:11:59.669756 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:11:59.670068 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:11:59.688174 9703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37365
I20260928 05:11:59.688196 9768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37365 every 8 connection(s)
I20260928 05:11:59.689523 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:11:59.693262 9769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:11:59.695719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9703
I20260928 05:11:59.695912 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:11:59.696673 388 external_mini_cluster.cc:1428] Setting key 14e23895930b13153d60f1563787e7c8
I20260928 05:11:59.705384 9769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e: Bootstrap starting.
I20260928 05:11:59.708878 9769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e: Neither blocks nor log segments found. Creating new log.
I20260928 05:11:59.709951 9769 log.cc:824] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:11:59.710139 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:11:59.698950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.0.97.62:37365)
0928 05:11:59.699923 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0928 05:11:59.699933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:11:59.699974 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:11:59.701526 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:11:59.701546 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:11:59.701991 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:11:59.702303 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:11:59.702898 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.704145 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.705938 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:11:59.706624 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:11:59.706764 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:11:59.707961 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:11:59.707968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:11:59.707979 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:11:59.708473 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:11:59.708544 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:11:59.708571 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:11:59.708654 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 05:11:59.708894 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 05:11:59.709168 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":352,"threads_started":1}
I20260928 05:11:59.712731 9769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e: No bootstrap required, opened a new log
I20260928 05:11:59.719899 9769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e11c57ecbb44659a62f653cd008f47e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37365 } }
I20260928 05:11:59.720278 9769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:11:59.720391 9769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e11c57ecbb44659a62f653cd008f47e, State: Initialized, Role: FOLLOWER
I20260928 05:11:59.721027 9769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e11c57ecbb44659a62f653cd008f47e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37365 } }
I20260928 05:11:59.721244 9769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:11:59.721393 9769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:11:59.721596 9769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:11:59.723750 9769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e11c57ecbb44659a62f653cd008f47e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37365 } }
I20260928 05:11:59.724161 9769 leader_election.cc:302] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e11c57ecbb44659a62f653cd008f47e; no voters:
I20260928 05:11:59.724695 9769 leader_election.cc:288] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:11:59.724833 9774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:11:59.725749 9774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [term 1 LEADER]: Becoming Leader. State: Replica: 0e11c57ecbb44659a62f653cd008f47e, State: Running, Role: LEADER
I20260928 05:11:59.726284 9774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e11c57ecbb44659a62f653cd008f47e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37365 } }
I20260928 05:11:59.726796 9769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:11:59.728924 9775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e11c57ecbb44659a62f653cd008f47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e11c57ecbb44659a62f653cd008f47e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37365 } } }
I20260928 05:11:59.729288 9775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:59.730124 9776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e11c57ecbb44659a62f653cd008f47e. Latest consensus state: current_term: 1 leader_uuid: "0e11c57ecbb44659a62f653cd008f47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e11c57ecbb44659a62f653cd008f47e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37365 } } }
I20260928 05:11:59.730384 9776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e [sys.catalog]: This master's current role is: LEADER
I20260928 05:11:59.732014 9783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:11:59.736734 9783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:11:59.745398 9783 catalog_manager.cc:1347] Generated new cluster ID: af6d2e0d7e5f4b1b9403f312be187447
I20260928 05:11:59.745515 9783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:11:59.764128 9783 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:11:59.765441 9783 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:11:59.773694 9783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e11c57ecbb44659a62f653cd008f47e: Generated new TSK 0
I20260928 05:11:59.774503 9783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:11:59.783360 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37365
--builtin_ntp_servers=127.0.97.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 05:12:00.131647 9794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:00.131904 9794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:00.132067 9794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:00.139230 9794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:00.139336 9794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:00.139503 9794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:12:00.148268 9794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:00.149490 9794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:00.151178 9794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:00.164321 9799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:00.164273 9802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:00.164286 9800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:00.165544 9794 server_base.cc:1034] running on GCE node
I20260928 05:12:00.166141 9794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:00.167158 9794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:00.168422 9794 hybrid_clock.cc:648] HybridClock initialized: now 1790572320168368 us; error 42 us; skew 500 ppm
I20260928 05:12:00.168778 9794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:00.174902 9794 webserver.cc:460] Webserver started at http://127.0.97.1:46729/ using document root <none> and password file <none>
I20260928 05:12:00.175673 9794 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:00.175837 9794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:00.176173 9794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:00.179152 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "ead97a9548c849c7b28380ba7e041262"
format_stamp: "Formatted at 2026-09-28 05:12:00 on dist-test-slave-6718"
server_key: "0d237ffde1af478685e514b20084aa56"
server_key_iv: "f6b6bc33728d4cc6c4866612790ad9ed"
server_key_version: "encryptionkey@0"
I20260928 05:12:00.179940 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ead97a9548c849c7b28380ba7e041262"
format_stamp: "Formatted at 2026-09-28 05:12:00 on dist-test-slave-6718"
server_key: "0d237ffde1af478685e514b20084aa56"
server_key_iv: "f6b6bc33728d4cc6c4866612790ad9ed"
server_key_version: "encryptionkey@0"
I20260928 05:12:00.184981 9794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:12:00.188405 9808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:00.189632 9794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 05:12:00.189810 9794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "ead97a9548c849c7b28380ba7e041262"
format_stamp: "Formatted at 2026-09-28 05:12:00 on dist-test-slave-6718"
server_key: "0d237ffde1af478685e514b20084aa56"
server_key_iv: "f6b6bc33728d4cc6c4866612790ad9ed"
server_key_version: "encryptionkey@0"
I20260928 05:12:00.190026 9794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:00.207823 9794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:00.208628 9794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:00.208899 9794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:00.209877 9794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:00.211359 9794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:00.211472 9794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:00.211604 9794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:00.211689 9794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:00.244168 9794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40931
I20260928 05:12:00.244218 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40931 every 8 connection(s)
I20260928 05:12:00.245462 9794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:12:00.248281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9794
I20260928 05:12:00.248485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:12:00.249267 388 external_mini_cluster.cc:1428] Setting key 270955d7cb856dacafcf3e982aae807c
I20260928 05:12:00.252170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37365
--builtin_ntp_servers=127.0.97.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 05:12:00.259464 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:00.247985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50029 (local address 127.0.97.62:37365)
0928 05:12:00.248500 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:00.248508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:00.249909 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 05:12:00.251885 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:00.251889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:00.251953 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:00.252037 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:00.253407 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.254101 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.256458 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.256852 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.256942 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:00.258577 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:00.258585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:00.258592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:00.258616 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:00.258665 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:00.258681 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:00.258762 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:00.259005 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 05:12:00.259099 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":251,"threads_started":1}
I20260928 05:12:00.260748 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:00.248659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37365 (local address 127.0.97.1:50029)
0928 05:12:00.249730 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:00.249782 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:00.251608 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:00.252307 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:00.252321 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:00.252446 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:00.253240 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:00.253268 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.254250 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.254258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:00.256324 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:00.256345 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.257066 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.257075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:00.257401 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:00.258198 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:12:00.258204 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:00.258256 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:12:00.258350 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:00.258784 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:00.258796 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:00.258806 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:12:00.258987 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 05:12:00.259404 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":293,"threads_started":1}
I20260928 05:12:00.262192 9921 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37365
I20260928 05:12:00.262485 9921 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:00.263370 9921 heartbeater.cc:511] Master 127.0.97.62:37365 requested a full tablet report, sending...
I20260928 05:12:00.265372 9734 ts_manager.cc:194] Registered new tserver with Master: ead97a9548c849c7b28380ba7e041262 (127.0.97.1:40931)
I20260928 05:12:00.266870 9734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50029
W20260928 05:12:00.576617 9925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:00.576869 9925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:00.577018 9925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:00.583971 9925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:00.584067 9925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:00.584244 9925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:12:00.592935 9925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:00.594175 9925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:00.595963 9925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:00.608589 9930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:00.608651 9931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:00.608675 9933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:00.617101 9925 server_base.cc:1034] running on GCE node
I20260928 05:12:00.617843 9925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:00.618919 9925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:00.620155 9925 hybrid_clock.cc:648] HybridClock initialized: now 1790572320620088 us; error 64 us; skew 500 ppm
I20260928 05:12:00.620487 9925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:00.623178 9925 webserver.cc:460] Webserver started at http://127.0.97.2:45623/ using document root <none> and password file <none>
I20260928 05:12:00.623890 9925 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:00.624028 9925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:00.624356 9925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:00.627590 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "b5f29fbbe165408187681502cac50c2b"
format_stamp: "Formatted at 2026-09-28 05:12:00 on dist-test-slave-6718"
server_key: "20a0a510c117011081e98654998695b1"
server_key_iv: "115669a6ac3033f167d01ef4581c7197"
server_key_version: "encryptionkey@0"
I20260928 05:12:00.628551 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b5f29fbbe165408187681502cac50c2b"
format_stamp: "Formatted at 2026-09-28 05:12:00 on dist-test-slave-6718"
server_key: "20a0a510c117011081e98654998695b1"
server_key_iv: "115669a6ac3033f167d01ef4581c7197"
server_key_version: "encryptionkey@0"
I20260928 05:12:00.633900 9925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:12:00.637328 9939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:00.638785 9925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:12:00.638967 9925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "b5f29fbbe165408187681502cac50c2b"
format_stamp: "Formatted at 2026-09-28 05:12:00 on dist-test-slave-6718"
server_key: "20a0a510c117011081e98654998695b1"
server_key_iv: "115669a6ac3033f167d01ef4581c7197"
server_key_version: "encryptionkey@0"
I20260928 05:12:00.639195 9925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:00.652209 9925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:00.653082 9925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:00.653367 9925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:00.654284 9925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:00.655750 9925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:00.655872 9925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:00.656010 9925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:00.656100 9925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:00.692808 9925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44203
I20260928 05:12:00.692848 10051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44203 every 8 connection(s)
I20260928 05:12:00.694235 9925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:12:00.697058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 9925
I20260928 05:12:00.697247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:12:00.697659 388 external_mini_cluster.cc:1428] Setting key 0a8a8f3aeb3d2b3aabc3ac7eb3acbf9b
I20260928 05:12:00.701305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37365
--builtin_ntp_servers=127.0.97.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 05:12:00.709255 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:00.698379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42329 (local address 127.0.97.62:37365)
0928 05:12:00.698832 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:00.698840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:00.698861 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:12:00.701036 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:00.701042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:00.701113 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:00.701213 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:00.702641 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.703762 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.706211 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.706669 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.706767 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:00.708361 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:00.708368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:00.708376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:00.708403 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:00.708442 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:00.708457 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:00.708567 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:00.708792 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 05:12:00.708897 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260928 05:12:00.710079 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:00.697544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37365 (local address 127.0.97.2:42329)
0928 05:12:00.698734 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:00.698779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:00.700740 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:00.701484 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:00.701498 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:00.701629 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:00.702430 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:00.702456 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.703929 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.703937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:00.706054 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:00.706071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:00.706888 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:00.706897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:00.707223 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:00.707964 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:12:00.707969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:00.708023 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:12:00.708126 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:00.708588 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:00.708596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:00.708601 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:12:00.708747 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 05:12:00.709027 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":326,"threads_started":1}
I20260928 05:12:00.711488 10052 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37365
I20260928 05:12:00.711794 10052 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:00.712527 10052 heartbeater.cc:511] Master 127.0.97.62:37365 requested a full tablet report, sending...
I20260928 05:12:00.713876 9734 ts_manager.cc:194] Registered new tserver with Master: b5f29fbbe165408187681502cac50c2b (127.0.97.2:44203)
I20260928 05:12:00.714753 9734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42329
W20260928 05:12:01.026216 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:01.026482 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:01.026706 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:01.034281 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:01.034375 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:01.034574 10055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:12:01.043756 10055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:01.044968 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:01.046783 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:01.059464 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:01.059527 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:01.059507 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:01.060266 10055 server_base.cc:1034] running on GCE node
I20260928 05:12:01.061031 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:01.062036 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:01.063237 10055 hybrid_clock.cc:648] HybridClock initialized: now 1790572321063185 us; error 46 us; skew 500 ppm
I20260928 05:12:01.063741 10055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:01.065958 10055 webserver.cc:460] Webserver started at http://127.0.97.3:44917/ using document root <none> and password file <none>
I20260928 05:12:01.066708 10055 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:01.066838 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:01.067157 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:01.069926 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "75edf5bbef144042af7505a75898afa7"
format_stamp: "Formatted at 2026-09-28 05:12:01 on dist-test-slave-6718"
server_key: "4b31cc1592695e3323657085aaffbb52"
server_key_iv: "7a213c9ce4ccf8234c0f3a39c9d3d68d"
server_key_version: "encryptionkey@0"
I20260928 05:12:01.070762 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "75edf5bbef144042af7505a75898afa7"
format_stamp: "Formatted at 2026-09-28 05:12:01 on dist-test-slave-6718"
server_key: "4b31cc1592695e3323657085aaffbb52"
server_key_iv: "7a213c9ce4ccf8234c0f3a39c9d3d68d"
server_key_version: "encryptionkey@0"
I20260928 05:12:01.075614 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:12:01.078821 10069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:01.080103 10055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:12:01.080276 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "75edf5bbef144042af7505a75898afa7"
format_stamp: "Formatted at 2026-09-28 05:12:01 on dist-test-slave-6718"
server_key: "4b31cc1592695e3323657085aaffbb52"
server_key_iv: "7a213c9ce4ccf8234c0f3a39c9d3d68d"
server_key_version: "encryptionkey@0"
I20260928 05:12:01.080487 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:01.091706 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:01.092434 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:01.092701 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:01.093686 10055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:01.095161 10055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:01.095278 10055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:01.095408 10055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:01.095497 10055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:01.127612 10055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46001
I20260928 05:12:01.127700 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46001 every 8 connection(s)
I20260928 05:12:01.128932 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:12:01.134591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10055
I20260928 05:12:01.134811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:12:01.135423 388 external_mini_cluster.cc:1428] Setting key 611be63fb8437419094f5aaf80d59178
I20260928 05:12:01.142058 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:01.131628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37269 (local address 127.0.97.62:37365)
0928 05:12:01.131809 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:01.131813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:01.133540 (+ 1727us) server_negotiation.cc:407] Connection header received
0928 05:12:01.135428 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:01.135434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:01.135518 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:01.135648 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:01.136911 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.137900 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.139695 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.140099 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.140195 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:01.141351 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:01.141356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:01.141359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:01.141381 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:01.141409 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:01.141420 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:01.141501 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:01.141734 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 05:12:01.141801 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:12:01.143195 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:01.132353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37365 (local address 127.0.97.3:37269)
0928 05:12:01.133380 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:01.133423 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:01.135169 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:01.135883 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:01.135898 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:01.136024 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:01.136715 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:01.136741 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.138054 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.138059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:01.139569 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:01.139581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.140275 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.140279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:01.140463 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:01.141052 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:12:01.141055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:01.141095 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:12:01.141185 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:01.141543 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:01.141551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:01.141558 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:12:01.141718 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 05:12:01.142062 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260928 05:12:01.144579 10182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37365
I20260928 05:12:01.144816 10182 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:01.145483 10182 heartbeater.cc:511] Master 127.0.97.62:37365 requested a full tablet report, sending...
I20260928 05:12:01.146814 9734 ts_manager.cc:194] Registered new tserver with Master: 75edf5bbef144042af7505a75898afa7 (127.0.97.3:46001)
I20260928 05:12:01.147552 9734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37269
I20260928 05:12:01.150926 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:12:01.156816 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:12:01.155198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.0.97.62:37365)
0928 05:12:01.155396 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:01.155405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:01.155794 (+ 389us) server_negotiation.cc:407] Connection header received
0928 05:12:01.155990 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:01.155995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:01.156071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:01.156168 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:01.156584 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60888: BlockingRecv error: recv got EOF from 127.0.0.1:60888 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:12:01.157274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9794
I20260928 05:12:01.165340 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:01.158276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.97.1:40931)
0928 05:12:01.158941 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:01.158947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:01.158981 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:12:01.159032 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:01.159038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:01.159228 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:01.159344 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:01.159701 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.160863 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.162241 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.162869 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.162972 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:01.163937 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:01.163945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:01.163957 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:01.164541 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:01.164585 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:01.164603 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:01.164691 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:01.164842 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 05:12:01.165014 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":391,"threads_started":1}
I20260928 05:12:01.166520 9916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:01.248881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9794
I20260928 05:12:01.264430 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9925
I20260928 05:12:01.272200 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:01.265310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.0.97.2:44203)
0928 05:12:01.265877 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:01.265884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:01.265910 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:12:01.265970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:01.265976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:01.266192 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:01.266320 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:01.266829 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.267811 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.269045 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.269619 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.269722 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:01.270830 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:01.270838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:01.270851 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:01.271466 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:01.271544 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:01.271564 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:01.271657 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:01.271815 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 05:12:01.271916 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":303,"threads_started":1}
I20260928 05:12:01.273391 10047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:01.353802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9925
I20260928 05:12:01.371353 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10055
I20260928 05:12:01.379609 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:01.372400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.0.97.3:46001)
0928 05:12:01.373012 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:01.373020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:01.373050 (+ 30us) server_negotiation.cc:407] Connection header received
0928 05:12:01.373117 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:01.373124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:01.373352 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:01.373523 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:01.373900 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.375152 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.376359 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:01.376951 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:01.377059 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:01.378009 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:01.378017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:01.378030 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:01.378753 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:01.378798 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:01.378816 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:01.378911 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:01.379064 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 05:12:01.379213 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":319,"threads_started":1}
I20260928 05:12:01.380903 10177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:01.461841 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10055
I20260928 05:12:01.478394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 9703
I20260928 05:12:01.479583 9764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:01.580456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 9703
2026-09-28T05:12:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2361 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T05:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:12:01Z Disabled control of system clock
I20260928 05:12:01.649142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42421
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:12:01.967767 10201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:01.968044 10201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:01.968127 10201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:01.975107 10201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:12:01.975210 10201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:01.975248 10201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:01.975296 10201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:12:01.975339 10201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:12:01.984019 10201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:01.985186 10201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:01.986933 10201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:01.998646 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:01.998580 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:01.998617 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:01.999459 10201 server_base.cc:1034] running on GCE node
I20260928 05:12:02.000190 10201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:02.001542 10201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:02.002739 10201 hybrid_clock.cc:648] HybridClock initialized: now 1790572322002672 us; error 70 us; skew 500 ppm
I20260928 05:12:02.003091 10201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:02.005404 10201 webserver.cc:460] Webserver started at http://127.0.97.62:41965/ using document root <none> and password file <none>
I20260928 05:12:02.006085 10201 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:02.006206 10201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:02.006582 10201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:02.009526 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "47655b31f02045a0887c42ced2812825"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "f2c85580307ac4ffd108bc55da1d85d3"
server_key_iv: "414ca35c333756e84b799442010d3973"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.010298 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "47655b31f02045a0887c42ced2812825"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "f2c85580307ac4ffd108bc55da1d85d3"
server_key_iv: "414ca35c333756e84b799442010d3973"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.015305 10201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:12:02.018726 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:02.020105 10201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:12:02.020291 10201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "47655b31f02045a0887c42ced2812825"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "f2c85580307ac4ffd108bc55da1d85d3"
server_key_iv: "414ca35c333756e84b799442010d3973"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.020507 10201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:02.026201 10201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:02.027034 10201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:02.027395 10201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:02.046743 10201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42421
I20260928 05:12:02.046787 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42421 every 8 connection(s)
I20260928 05:12:02.048077 10201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:12:02.051729 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:02.058269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10201
I20260928 05:12:02.058461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:12:02.059058 388 external_mini_cluster.cc:1428] Setting key d8e27faa1a50eed5fb22967ff037aff9
I20260928 05:12:02.063419 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825: Bootstrap starting.
I20260928 05:12:02.066898 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:02.067368 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:02.061112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.0.97.62:42421)
0928 05:12:02.062094 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:02.062105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:02.062146 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:12:02.063884 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:02.063905 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:02.064290 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:02.064685 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:02.065732 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:02.065738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:02.065749 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:02.065762 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:02.065818 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:02.065833 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:02.066053 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:02.066182 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:12:02.066406 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":360,"threads_started":1}
I20260928 05:12:02.068063 10267 log.cc:824] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:02.071019 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825: No bootstrap required, opened a new log
I20260928 05:12:02.079046 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47655b31f02045a0887c42ced2812825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42421 } }
I20260928 05:12:02.079435 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:02.079563 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47655b31f02045a0887c42ced2812825, State: Initialized, Role: FOLLOWER
I20260928 05:12:02.080070 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47655b31f02045a0887c42ced2812825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42421 } }
I20260928 05:12:02.080276 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:12:02.080374 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:12:02.080511 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:02.082887 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47655b31f02045a0887c42ced2812825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42421 } }
I20260928 05:12:02.083576 10267 leader_election.cc:302] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47655b31f02045a0887c42ced2812825; no voters:
I20260928 05:12:02.084257 10267 leader_election.cc:288] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:12:02.084369 10272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:12:02.085145 10272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [term 1 LEADER]: Becoming Leader. State: Replica: 47655b31f02045a0887c42ced2812825, State: Running, Role: LEADER
I20260928 05:12:02.085687 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47655b31f02045a0887c42ced2812825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42421 } }
I20260928 05:12:02.086611 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:12:02.088586 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47655b31f02045a0887c42ced2812825. Latest consensus state: current_term: 1 leader_uuid: "47655b31f02045a0887c42ced2812825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47655b31f02045a0887c42ced2812825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42421 } } }
I20260928 05:12:02.088621 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47655b31f02045a0887c42ced2812825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47655b31f02045a0887c42ced2812825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42421 } } }
I20260928 05:12:02.088994 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [sys.catalog]: This master's current role is: LEADER
I20260928 05:12:02.088989 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825 [sys.catalog]: This master's current role is: LEADER
I20260928 05:12:02.091998 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:12:02.098505 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:12:02.107585 10279 catalog_manager.cc:1347] Generated new cluster ID: 5458b730d32e4dc3b0fceac3cbd748a6
I20260928 05:12:02.107772 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:12:02.141783 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:12:02.142405 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:12:02.154630 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47655b31f02045a0887c42ced2812825: Generated new TSK 0
I20260928 05:12:02.155730 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:12:02.172926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42421
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 05:12:02.496070 10291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:02.496387 10291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:02.496546 10291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:02.503757 10291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:02.503901 10291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:02.504082 10291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:12:02.513371 10291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:02.514753 10291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:02.516461 10291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:02.529642 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:02.529603 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:02.529690 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:02.530387 10291 server_base.cc:1034] running on GCE node
I20260928 05:12:02.530982 10291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:02.531876 10291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:02.533098 10291 hybrid_clock.cc:648] HybridClock initialized: now 1790572322533063 us; error 61 us; skew 500 ppm
I20260928 05:12:02.533406 10291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:02.535732 10291 webserver.cc:460] Webserver started at http://127.0.97.1:35979/ using document root <none> and password file <none>
I20260928 05:12:02.536397 10291 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:02.536530 10291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:02.536851 10291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:02.539698 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "b4647a0f90084da0b6d2cf1403f39422"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "149d85c61dff4d18e77588d3d2f8170d"
server_key_iv: "6f74f02010af0eb3878cca50c932262f"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.540472 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b4647a0f90084da0b6d2cf1403f39422"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "149d85c61dff4d18e77588d3d2f8170d"
server_key_iv: "6f74f02010af0eb3878cca50c932262f"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.545367 10291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 05:12:02.548677 10305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:02.549930 10291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:12:02.550108 10291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "b4647a0f90084da0b6d2cf1403f39422"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "149d85c61dff4d18e77588d3d2f8170d"
server_key_iv: "6f74f02010af0eb3878cca50c932262f"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.550331 10291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:02.555745 10291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:02.556473 10291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:02.556738 10291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:02.557587 10291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:02.558959 10291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:02.559077 10291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:02.559206 10291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:02.559289 10291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:02.592049 10291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40165
I20260928 05:12:02.592100 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40165 every 8 connection(s)
I20260928 05:12:02.593565 10291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:12:02.596342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10291
I20260928 05:12:02.596571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:12:02.597259 388 external_mini_cluster.cc:1428] Setting key 3eb7afec37d56732cd5fa2f9f8d23d27
I20260928 05:12:02.600605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42421
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 05:12:02.603441 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:02.596406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33387 (local address 127.0.97.62:42421)
0928 05:12:02.596892 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:02.596898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:02.598755 (+ 1857us) server_negotiation.cc:407] Connection header received
0928 05:12:02.600984 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:02.600991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:02.601066 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:02.601174 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:02.602601 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:02.602609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:02.602616 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:02.602638 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:02.602689 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:02.602707 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:02.602826 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:02.603016 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 05:12:02.603077 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260928 05:12:02.604306 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:02.597106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42421 (local address 127.0.97.1:33387)
0928 05:12:02.598589 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:02.598639 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:02.600569 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:02.601425 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:02.601434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:02.601532 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:02.602253 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:12:02.602255 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:02.602294 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:12:02.602383 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:02.602809 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:02.602817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:02.602824 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:12:02.603053 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 05:12:02.603259 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260928 05:12:02.605930 10418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42421
I20260928 05:12:02.606261 10418 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:02.606943 10418 heartbeater.cc:511] Master 127.0.97.62:42421 requested a full tablet report, sending...
I20260928 05:12:02.609388 10232 ts_manager.cc:194] Registered new tserver with Master: b4647a0f90084da0b6d2cf1403f39422 (127.0.97.1:40165)
W20260928 05:12:02.922866 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:02.923142 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:02.923229 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:02.930408 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:02.930512 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:02.930686 10422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:12:02.939911 10422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:02.941092 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:02.942816 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:02.956166 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:02.956207 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:02.956526 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:02.957284 10422 server_base.cc:1034] running on GCE node
I20260928 05:12:02.957890 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:02.958889 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:02.960122 10422 hybrid_clock.cc:648] HybridClock initialized: now 1790572322960076 us; error 52 us; skew 500 ppm
I20260928 05:12:02.960448 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:02.962760 10422 webserver.cc:460] Webserver started at http://127.0.97.2:46765/ using document root <none> and password file <none>
I20260928 05:12:02.963461 10422 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:02.963598 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:02.963912 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:02.966873 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "002047ff936245898ec6987fc336ebe0"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "61715509057466de6ac03e7ed7dbbe50"
server_key_iv: "770c7b808ed44e3dd0ac8726bcaff27f"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.967705 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "002047ff936245898ec6987fc336ebe0"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "61715509057466de6ac03e7ed7dbbe50"
server_key_iv: "770c7b808ed44e3dd0ac8726bcaff27f"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.973160 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:12:02.977017 10436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:02.978353 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 05:12:02.978585 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "002047ff936245898ec6987fc336ebe0"
format_stamp: "Formatted at 2026-09-28 05:12:02 on dist-test-slave-6718"
server_key: "61715509057466de6ac03e7ed7dbbe50"
server_key_iv: "770c7b808ed44e3dd0ac8726bcaff27f"
server_key_version: "encryptionkey@0"
I20260928 05:12:02.978832 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:02.984279 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:02.985030 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:02.985299 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:02.986150 10422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:02.987613 10422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:02.987737 10422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:02.987857 10422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:02.987941 10422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:03.020041 10422 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35501
I20260928 05:12:03.020090 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35501 every 8 connection(s)
I20260928 05:12:03.021350 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:12:03.024443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10422
I20260928 05:12:03.024633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:12:03.025149 388 external_mini_cluster.cc:1428] Setting key 4b5b7f232f5e4cf440ea1454fdf1947a
I20260928 05:12:03.027640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42421
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 05:12:03.029825 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.023793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33087 (local address 127.0.97.62:42421)
0928 05:12:03.023996 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:03.024004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:03.025793 (+ 1789us) server_negotiation.cc:407] Connection header received
0928 05:12:03.027605 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:03.027610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:03.027674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:03.027795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:03.029108 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:03.029113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:03.029118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:03.029136 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:03.029173 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:03.029184 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:03.029317 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:03.029497 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 05:12:03.029560 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:12:03.030783 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.024438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42421 (local address 127.0.97.2:33087)
0928 05:12:03.025613 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:03.025664 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:03.027353 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:03.028018 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:03.028026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:03.028122 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:03.028827 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:12:03.028830 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:03.028867 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:12:03.028948 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:03.029280 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:03.029291 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:03.029301 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:12:03.029578 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 05:12:03.029831 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":428,"threads_started":1}
I20260928 05:12:03.032096 10549 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42421
I20260928 05:12:03.032387 10549 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:03.032908 10549 heartbeater.cc:511] Master 127.0.97.62:42421 requested a full tablet report, sending...
I20260928 05:12:03.034300 10232 ts_manager.cc:194] Registered new tserver with Master: 002047ff936245898ec6987fc336ebe0 (127.0.97.2:35501)
W20260928 05:12:03.345353 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:03.345620 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:03.345763 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:03.352969 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:03.353061 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:03.353241 10552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:12:03.362325 10552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:03.363667 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:03.365348 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:03.378490 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:03.378571 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:03.378521 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:03.379402 10552 server_base.cc:1034] running on GCE node
I20260928 05:12:03.379971 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:03.380920 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:03.382117 10552 hybrid_clock.cc:648] HybridClock initialized: now 1790572323382072 us; error 52 us; skew 500 ppm
I20260928 05:12:03.382458 10552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:12:03.384779 10552 webserver.cc:460] Webserver started at http://127.0.97.3:32859/ using document root <none> and password file <none>
I20260928 05:12:03.385462 10552 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:03.385576 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:03.385890 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:03.388911 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "ea8e65f8bde24ce48346e708688bcbc8"
format_stamp: "Formatted at 2026-09-28 05:12:03 on dist-test-slave-6718"
server_key: "0e337d8d84926ba16d9c33d9d3a360bd"
server_key_iv: "da0dc4489723e1dc65a37bbe85634c7a"
server_key_version: "encryptionkey@0"
I20260928 05:12:03.389711 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ea8e65f8bde24ce48346e708688bcbc8"
format_stamp: "Formatted at 2026-09-28 05:12:03 on dist-test-slave-6718"
server_key: "0e337d8d84926ba16d9c33d9d3a360bd"
server_key_iv: "da0dc4489723e1dc65a37bbe85634c7a"
server_key_version: "encryptionkey@0"
I20260928 05:12:03.394806 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:12:03.398221 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:03.399660 10552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:12:03.399829 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "ea8e65f8bde24ce48346e708688bcbc8"
format_stamp: "Formatted at 2026-09-28 05:12:03 on dist-test-slave-6718"
server_key: "0e337d8d84926ba16d9c33d9d3a360bd"
server_key_iv: "da0dc4489723e1dc65a37bbe85634c7a"
server_key_version: "encryptionkey@0"
I20260928 05:12:03.400053 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:03.405440 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:03.406194 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:03.406451 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:03.407344 10552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:03.408668 10552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:03.408787 10552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:03.408915 10552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:03.409001 10552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:03.441677 10552 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38213
I20260928 05:12:03.441725 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38213 every 8 connection(s)
I20260928 05:12:03.442992 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:12:03.450212 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10552
I20260928 05:12:03.450376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:12:03.450981 388 external_mini_cluster.cc:1428] Setting key 241957a7aeb8418b47b619f3f9894a97
I20260928 05:12:03.451581 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.445653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58231 (local address 127.0.97.62:42421)
0928 05:12:03.445821 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:03.445825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:03.447343 (+ 1518us) server_negotiation.cc:407] Connection header received
0928 05:12:03.449001 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:03.449006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:03.449073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:03.449197 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:03.450707 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:03.450713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:03.450718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:03.450739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:03.450775 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:03.450789 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:03.450919 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:03.451214 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 05:12:03.451283 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:12:03.452628 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.446214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42421 (local address 127.0.97.3:58231)
0928 05:12:03.447182 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:03.447222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:03.448734 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:03.449421 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:03.449430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:03.449523 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:03.450274 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:12:03.450276 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:03.450314 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:12:03.450459 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:03.450929 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:03.450939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:03.450946 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 05:12:03.451279 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 05:12:03.451608 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":343,"threads_started":1}
I20260928 05:12:03.454097 10679 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42421
I20260928 05:12:03.454393 10679 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:03.455084 10679 heartbeater.cc:511] Master 127.0.97.62:42421 requested a full tablet report, sending...
I20260928 05:12:03.456457 10232 ts_manager.cc:194] Registered new tserver with Master: ea8e65f8bde24ce48346e708688bcbc8 (127.0.97.3:38213)
I20260928 05:12:03.466010 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 05:12:03.471974 10420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:12:03.470550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.0.97.62:42421)
0928 05:12:03.470705 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:03.470709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:03.471177 (+ 468us) server_negotiation.cc:407] Connection header received
0928 05:12:03.471300 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:03.471304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:03.471352 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:03.471431 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:03.471738 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45510: BlockingRecv error: recv got EOF from 127.0.0.1:45510 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:12:03.472638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10291
I20260928 05:12:03.476078 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.473624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.0.97.1:40165)
0928 05:12:03.474133 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:03.474140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:03.474165 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:12:03.474218 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:03.474224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:03.474438 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:03.474643 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:03.475470 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:03.475477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:03.475527 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:03.475547 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:03.475589 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:03.475604 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:03.475705 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:03.475784 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 05:12:03.475855 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":251,"threads_started":1}
I20260928 05:12:03.477185 10413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:03.562036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10291
I20260928 05:12:03.578943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10422
I20260928 05:12:03.582501 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.579861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.97.2:35501)
0928 05:12:03.580337 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:03.580344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:03.580369 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:12:03.580432 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:03.580438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:03.580660 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:03.580832 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:03.581837 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:03.581843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:03.581855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:03.581866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:03.581903 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:03.581916 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:03.582067 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:03.582145 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 05:12:03.582241 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260928 05:12:03.584023 10544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:03.668948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10422
I20260928 05:12:03.684818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10552
I20260928 05:12:03.688725 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:03.685808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.97.3:38213)
0928 05:12:03.686444 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:03.686452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:03.686481 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:12:03.686643 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:03.686655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:03.686882 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:03.687007 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:03.687929 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:03.687935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:03.687947 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:12:03.687962 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:03.688000 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:12:03.688012 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:03.688144 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:03.688296 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 05:12:03.688409 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
I20260928 05:12:03.689886 10674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:03.770964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10552
I20260928 05:12:03.786338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10201
I20260928 05:12:03.787619 10262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:03.888674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10201
2026-09-28T05:12:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2308 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:12:03 dist-test-slave-6718 krb5kdc[10697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:12:03 dist-test-slave-6718 krb5kdc[10697](info): set up 2 sockets
Sep 28 05:12:03 dist-test-slave-6718 krb5kdc[10697](info): commencing operation
krb5kdc: starting...
W20260928 05:12:05.044535 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:12:05 dist-test-slave-6718 krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:12:05.270781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45279
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:12:05.592857 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:05.593181 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:05.593344 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:05.600627 10710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:12:05.600769 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:05.600872 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:05.600972 10710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:12:05.601054 10710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:12:05.610713 10710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:05.612018 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:05.613297 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:05.624315 10718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:05.624379 10716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:05.624405 10715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:05.625429 10710 server_base.cc:1034] running on GCE node
I20260928 05:12:05.626232 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:05.627691 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:05.628896 10710 hybrid_clock.cc:648] HybridClock initialized: now 1790572325628855 us; error 28 us; skew 500 ppm
I20260928 05:12:05.629202 10710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:05 dist-test-slave-6718 krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:05.636255 10710 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:12:05.637676 10710 webserver.cc:460] Webserver started at http://127.0.97.62:35347/ using document root <none> and password file <none>
I20260928 05:12:05.638306 10710 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:05.638401 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:05.638753 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:05.641583 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "e69271b846244272ad3aec3ad75c9ad7"
format_stamp: "Formatted at 2026-09-28 05:12:05 on dist-test-slave-6718"
server_key: "7181c640a54d4fb1951121695ff72e2c"
server_key_iv: "356bcc1200da37d22c6699d3690ac466"
server_key_version: "encryptionkey@0"
I20260928 05:12:05.642259 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "e69271b846244272ad3aec3ad75c9ad7"
format_stamp: "Formatted at 2026-09-28 05:12:05 on dist-test-slave-6718"
server_key: "7181c640a54d4fb1951121695ff72e2c"
server_key_iv: "356bcc1200da37d22c6699d3690ac466"
server_key_version: "encryptionkey@0"
I20260928 05:12:05.647221 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:12:05.650353 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:05.651692 10710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:12:05.651863 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "e69271b846244272ad3aec3ad75c9ad7"
format_stamp: "Formatted at 2026-09-28 05:12:05 on dist-test-slave-6718"
server_key: "7181c640a54d4fb1951121695ff72e2c"
server_key_iv: "356bcc1200da37d22c6699d3690ac466"
server_key_version: "encryptionkey@0"
I20260928 05:12:05.652041 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:05.674009 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:05.677873 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:05.678151 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:05.696256 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45279
I20260928 05:12:05.696339 10776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45279 every 8 connection(s)
I20260928 05:12:05.697556 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:12:05.701227 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:05.702003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10710
I20260928 05:12:05.702210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:12:05.702929 388 external_mini_cluster.cc:1428] Setting key 5babec6a8f67659bbf3b0b4375dd0406
I20260928 05:12:05.715067 10777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: Bootstrap starting.
Sep 28 05:12:05 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:05.718901 10777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:05.720340 10777 log.cc:824] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:05.723567 10777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: No bootstrap required, opened a new log
I20260928 05:12:05.726109 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:05.704828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.62:45279)
0928 05:12:05.705578 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:05.705590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:05.705630 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:12:05.707251 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:05.707273 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:05.707679 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:05.708005 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:05.709754 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:05.711100 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:05.712967 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:05.713634 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:05.713760 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:05.718209 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:05.718219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:05.718237 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:05.718280 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:05.721910 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:05.722576 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:05.722584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:05.722591 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:05.722788 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:05.723114 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:05.723119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:05.723121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:05.723777 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:05.724072 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:05.724386 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 05:12:05.724781 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":342,"threads_started":1}
I20260928 05:12:05.732434 10777 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69271b846244272ad3aec3ad75c9ad7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } }
I20260928 05:12:05.732748 10777 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:05.732816 10777 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69271b846244272ad3aec3ad75c9ad7, State: Initialized, Role: FOLLOWER
I20260928 05:12:05.733258 10777 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69271b846244272ad3aec3ad75c9ad7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } }
I20260928 05:12:05.733460 10777 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:12:05.733615 10777 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:12:05.733804 10777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:05.736814 10777 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69271b846244272ad3aec3ad75c9ad7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } }
I20260928 05:12:05.737500 10777 leader_election.cc:302] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69271b846244272ad3aec3ad75c9ad7; no voters:
I20260928 05:12:05.738304 10777 leader_election.cc:288] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:12:05.738510 10782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:12:05.741009 10777 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:12:05.744197 10782 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [term 1 LEADER]: Becoming Leader. State: Replica: e69271b846244272ad3aec3ad75c9ad7, State: Running, Role: LEADER
I20260928 05:12:05.744966 10782 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69271b846244272ad3aec3ad75c9ad7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } }
I20260928 05:12:05.748409 10783 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69271b846244272ad3aec3ad75c9ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69271b846244272ad3aec3ad75c9ad7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } }
I20260928 05:12:05.748968 10783 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:12:05.753680 10784 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69271b846244272ad3aec3ad75c9ad7. Latest consensus state: current_term: 1 leader_uuid: "e69271b846244272ad3aec3ad75c9ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69271b846244272ad3aec3ad75c9ad7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } }
I20260928 05:12:05.754012 10784 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7 [sys.catalog]: This master's current role is: LEADER
W20260928 05:12:05.762521 10798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:12:05.762678 10798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:12:05.763700 10789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:12:05.766067 10789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:12:05.776722 10789 catalog_manager.cc:1347] Generated new cluster ID: 008d7dcc87cf4d2a905cc78a57da98b0
I20260928 05:12:05.776837 10789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:12:05.798607 10789 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:12:05.800606 10789 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:12:05.816706 10789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: Generated new TSK 0
I20260928 05:12:05.817744 10789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:12:05.878075 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45279
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:12:06.213176 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:06.213497 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:06.213706 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:06.220928 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:06.221069 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:06.221252 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:12:06.231067 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:06.232331 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:06.233664 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:06.247103 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:06.247074 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:06.247128 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:06.248265 10803 server_base.cc:1034] running on GCE node
I20260928 05:12:06.248908 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:06.249922 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:06.251143 10803 hybrid_clock.cc:648] HybridClock initialized: now 1790572326251076 us; error 60 us; skew 500 ppm
I20260928 05:12:06.251453 10803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:06 dist-test-slave-6718 krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:06.254511 10803 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:12:06.256138 10803 webserver.cc:460] Webserver started at http://127.0.97.1:43755/ using document root <none> and password file <none>
I20260928 05:12:06.256846 10803 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:06.256985 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:06.257280 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:06.260265 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "400b3681712c471d94b41fcf3bf27be3"
format_stamp: "Formatted at 2026-09-28 05:12:06 on dist-test-slave-6718"
server_key: "0dbb87ce41e5551a9006356bd5d42363"
server_key_iv: "d69b8781102b744ae8aab0595c4ebcad"
server_key_version: "encryptionkey@0"
I20260928 05:12:06.260999 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "400b3681712c471d94b41fcf3bf27be3"
format_stamp: "Formatted at 2026-09-28 05:12:06 on dist-test-slave-6718"
server_key: "0dbb87ce41e5551a9006356bd5d42363"
server_key_iv: "d69b8781102b744ae8aab0595c4ebcad"
server_key_version: "encryptionkey@0"
I20260928 05:12:06.265935 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:12:06.269307 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:06.270663 10803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:12:06.270826 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "400b3681712c471d94b41fcf3bf27be3"
format_stamp: "Formatted at 2026-09-28 05:12:06 on dist-test-slave-6718"
server_key: "0dbb87ce41e5551a9006356bd5d42363"
server_key_iv: "d69b8781102b744ae8aab0595c4ebcad"
server_key_version: "encryptionkey@0"
I20260928 05:12:06.271013 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:06.300005 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:06.304209 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:06.304510 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:06.305476 10803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:06.306816 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:06.306921 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:06.307068 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:06.307147 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:06.339965 10803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33969
I20260928 05:12:06.340013 10930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33969 every 8 connection(s)
I20260928 05:12:06.341290 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:12:06.343101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10803
I20260928 05:12:06.343286 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:12:06.344135 388 external_mini_cluster.cc:1428] Setting key 2791ade46bcf7f30ba2c1f41fffe0949
Sep 28 05:12:06 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:06.361647 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:06.344262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51143 (local address 127.0.97.62:45279)
0928 05:12:06.344743 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:06.344749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:06.346607 (+ 1858us) server_negotiation.cc:407] Connection header received
0928 05:12:06.347663 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:06.347669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:06.347750 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:06.347876 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:06.349295 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.350010 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.351601 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.352019 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.352150 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:06.355802 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:06.355811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:06.355815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:06.355846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:06.358495 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:06.359330 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:06.359334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:06.359336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:06.359455 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:06.359891 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:06.359902 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:06.359905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:06.360164 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:06.360314 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:06.361067 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 05:12:06.361277 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":242,"threads_started":1}
I20260928 05:12:06.362682 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:06.344859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.97.1:51143)
0928 05:12:06.345628 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:06.345673 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:06.347413 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:06.348112 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:06.348121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:06.348589 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:06.349134 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:06.349151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.350168 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.350174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:06.351461 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:06.351472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.352249 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.352258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:06.352486 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:06.353154 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:12:06.353184 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:06.355511 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:06.358713 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:06.358722 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:06.358740 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:06.359156 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:06.359569 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:06.359574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:06.359578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:06.359771 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:06.360293 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:06.360301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:06.360669 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 05:12:06.360938 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 05:12:06.361464 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":348,"threads_started":1}
I20260928 05:12:06.364315 10931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45279
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:12:06.364629 10931 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:06.365330 10931 heartbeater.cc:511] Master 127.0.97.62:45279 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:12:06.367647 10742 ts_manager.cc:194] Registered new tserver with Master: 400b3681712c471d94b41fcf3bf27be3 (127.0.97.1:33969)
I20260928 05:12:06.369036 10742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:12:06.393771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45279
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:12:06.715579 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:06.715844 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:06.715997 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:06.723356 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:06.723456 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:06.723666 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:12:06.733332 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:06.734627 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:06.735951 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:06.748924 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:06.749012 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:06.749034 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:06.750005 10937 server_base.cc:1034] running on GCE node
I20260928 05:12:06.750737 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:06.751722 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:06.752952 10937 hybrid_clock.cc:648] HybridClock initialized: now 1790572326752909 us; error 40 us; skew 500 ppm
I20260928 05:12:06.753273 10937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:06 dist-test-slave-6718 krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:06.756740 10937 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:12:06.758203 10937 webserver.cc:460] Webserver started at http://127.0.97.2:38149/ using document root <none> and password file <none>
I20260928 05:12:06.759042 10937 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:06.759189 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:06.759491 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:06.762341 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "d9027884ce104a19a942a0e301184937"
format_stamp: "Formatted at 2026-09-28 05:12:06 on dist-test-slave-6718"
server_key: "1950908fd9fd70cb4d4fd0aa03024fde"
server_key_iv: "d021605bfb00839dff435cc88fada0e6"
server_key_version: "encryptionkey@0"
I20260928 05:12:06.763247 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d9027884ce104a19a942a0e301184937"
format_stamp: "Formatted at 2026-09-28 05:12:06 on dist-test-slave-6718"
server_key: "1950908fd9fd70cb4d4fd0aa03024fde"
server_key_iv: "d021605bfb00839dff435cc88fada0e6"
server_key_version: "encryptionkey@0"
I20260928 05:12:06.768160 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:12:06.771543 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:06.772943 10937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:12:06.773120 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "d9027884ce104a19a942a0e301184937"
format_stamp: "Formatted at 2026-09-28 05:12:06 on dist-test-slave-6718"
server_key: "1950908fd9fd70cb4d4fd0aa03024fde"
server_key_iv: "d021605bfb00839dff435cc88fada0e6"
server_key_version: "encryptionkey@0"
I20260928 05:12:06.773329 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:06.801054 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:06.804986 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:06.805279 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:06.806211 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:06.807502 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:06.807617 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:06.807767 10937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:06.807839 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:06.840282 10937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33215
I20260928 05:12:06.840330 11064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33215 every 8 connection(s)
I20260928 05:12:06.841638 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:12:06.844591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 10937
I20260928 05:12:06.844771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:12:06.845206 388 external_mini_cluster.cc:1428] Setting key 337abaa5f3d75ae16765fa80292865f4
Sep 28 05:12:06 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:06.862210 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:06.844202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43487 (local address 127.0.97.62:45279)
0928 05:12:06.844401 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:06.844408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:06.845753 (+ 1345us) server_negotiation.cc:407] Connection header received
0928 05:12:06.847652 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:06.847659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:06.847744 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:06.847848 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:06.849251 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.850318 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.851874 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.852637 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.852777 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:06.856831 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:06.856836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:06.856839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:06.856868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:06.859361 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:06.860068 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:06.860075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:06.860083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:06.860192 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:06.860595 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:06.860599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:06.860613 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:06.860861 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:06.861008 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:06.861664 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 05:12:06.861874 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:12:06.863242 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:06.844762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.97.2:43487)
0928 05:12:06.845592 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:06.845645 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:06.847397 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:06.848030 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:06.848037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:06.848523 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:06.849078 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:06.849094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.850471 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.850475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:06.851739 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:06.851751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:06.852829 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:06.852833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:06.853029 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:06.853698 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:12:06.853719 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:06.856611 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:06.859494 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:06.859503 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:06.859517 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:06.859913 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:06.860297 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:06.860303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:06.860306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:06.860480 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:06.860982 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:06.860989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:06.861315 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 05:12:06.861562 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 05:12:06.862038 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":347,"threads_started":1}
I20260928 05:12:06.864723 11065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45279
I20260928 05:12:06.865008 11065 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:06.865675 11065 heartbeater.cc:511] Master 127.0.97.62:45279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 05:12:06.867156 10742 ts_manager.cc:194] Registered new tserver with Master: d9027884ce104a19a942a0e301184937 (127.0.97.2:33215)
I20260928 05:12:06.867995 10742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:12:06.894809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45279
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:12:07.212199 11070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:07.212461 11070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:07.212553 11070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:07.219525 11070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:07.219623 11070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:07.219738 11070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:12:07.228984 11070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:07.230162 11070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:07.231446 11070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:07.245739 11075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:07.245730 11076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:07.245774 11078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:07.246555 11070 server_base.cc:1034] running on GCE node
I20260928 05:12:07.247171 11070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:07.248087 11070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:07.249276 11070 hybrid_clock.cc:648] HybridClock initialized: now 1790572327249212 us; error 57 us; skew 500 ppm
I20260928 05:12:07.249583 11070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:07 dist-test-slave-6718 krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:07.252626 11070 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:12:07.254072 11070 webserver.cc:460] Webserver started at http://127.0.97.3:34029/ using document root <none> and password file <none>
I20260928 05:12:07.254830 11070 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:07.254984 11070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:07.255326 11070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:12:07.258226 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "4823efc7ff084243816ee5ead34edb95"
format_stamp: "Formatted at 2026-09-28 05:12:07 on dist-test-slave-6718"
server_key: "09fcbd33e0975f57de04255e63613757"
server_key_iv: "d5d0a45e09a75175dec522282745f5c8"
server_key_version: "encryptionkey@0"
I20260928 05:12:07.259109 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4823efc7ff084243816ee5ead34edb95"
format_stamp: "Formatted at 2026-09-28 05:12:07 on dist-test-slave-6718"
server_key: "09fcbd33e0975f57de04255e63613757"
server_key_iv: "d5d0a45e09a75175dec522282745f5c8"
server_key_version: "encryptionkey@0"
I20260928 05:12:07.263861 11070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 05:12:07.267061 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:07.268337 11070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:12:07.268508 11070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "4823efc7ff084243816ee5ead34edb95"
format_stamp: "Formatted at 2026-09-28 05:12:07 on dist-test-slave-6718"
server_key: "09fcbd33e0975f57de04255e63613757"
server_key_iv: "d5d0a45e09a75175dec522282745f5c8"
server_key_version: "encryptionkey@0"
I20260928 05:12:07.268707 11070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:07.287375 11070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:07.291278 11070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:07.291572 11070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:07.292475 11070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:07.293730 11070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:07.293829 11070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:07.293982 11070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:07.294054 11070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:07.327085 11070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35609
I20260928 05:12:07.327175 11197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35609 every 8 connection(s)
I20260928 05:12:07.328336 11070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:12:07.336254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 11070
I20260928 05:12:07.336416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:12:07.336827 388 external_mini_cluster.cc:1428] Setting key 23d69719cabd757df42e0f74494b1d7d
Sep 28 05:12:07 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:07.348163 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.330789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47861 (local address 127.0.97.62:45279)
0928 05:12:07.330996 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.331002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.332188 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 05:12:07.333842 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.333847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.333913 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.334006 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:07.335522 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.336368 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.338155 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.338731 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.338861 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.342384 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:07.342389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:07.342392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:07.342430 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:07.345235 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.345926 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.345933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.345939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.346071 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.346483 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.346488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.346492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.346804 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:07.346953 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.347608 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.347811 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:12:07.349038 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.331319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.97.3:47861)
0928 05:12:07.332021 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:07.332065 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:07.333579 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:07.334210 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:07.334218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:07.334791 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:07.335342 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.335359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.336545 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.336553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.337967 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.337980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.338951 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.338960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.339225 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.340024 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:12:07.340053 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:07.342185 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:07.345377 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:07.345385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:07.345399 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:07.345805 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:07.346179 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:07.346185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:07.346189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:07.346364 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:07.346928 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:07.346936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:07.347253 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 05:12:07.347521 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 05:12:07.348017 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":279,"threads_started":1}
I20260928 05:12:07.350378 11198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45279
I20260928 05:12:07.350718 11198 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:07.351532 11198 heartbeater.cc:511] Master 127.0.97.62:45279 requested a full tablet report, sending...
I20260928 05:12:07.352933 10742 ts_manager.cc:194] Registered new tserver with Master: 4823efc7ff084243816ee5ead34edb95 (127.0.97.3:35609)
I20260928 05:12:07.353787 10742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47861
I20260928 05:12:07.363999 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:12:07.371775 10931 heartbeater.cc:503] Master 127.0.97.62:45279 was elected leader, sending a full tablet report...
I20260928 05:12:07.380342 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.368178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.0.97.62:45279)
0928 05:12:07.368392 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.368399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.368754 (+ 355us) server_negotiation.cc:407] Connection header received
0928 05:12:07.368963 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.368967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.369035 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.369137 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:07.370431 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.371192 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.372677 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.373276 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.373430 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.375650 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:07.375655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:07.375658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:07.375705 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:07.378088 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.378792 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.378796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.378799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.378910 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.379220 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.379223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.379225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.379499 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:07.379681 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.379949 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.380110 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:12:07.384138 10742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:12:07.390734 10742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:12:07.413707 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.401751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.97.2:33215)
0928 05:12:07.402348 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.402356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.403017 (+ 661us) server_negotiation.cc:407] Connection header received
0928 05:12:07.403421 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.403429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.403679 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.403840 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:07.404971 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.406764 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.410003 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.411850 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.411987 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.412246 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.413083 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.413314 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":325,"threads_started":1}
I20260928 05:12:07.414669 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.401519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33215 (local address 127.0.0.1:53454)
0928 05:12:07.402878 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:07.402923 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:07.403170 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:07.404205 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:07.404215 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:07.404256 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:07.404745 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.404760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.406925 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.406929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.409836 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.409849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.412476 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.412484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.412758 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.412826 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 05:12:07.414056 (+ 1230us) client_negotiation.cc:240] Negotiation successful
0928 05:12:07.414241 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":966,"threads_started":1}
I20260928 05:12:07.421295 11000 tablet_service.cc:1467] Processing CreateTablet for tablet adcfcae5fcc34b06b10f28806910de10 (DEFAULT_TABLE table=test-table [id=ceb0bfe71bea41fa9b430ed36e7a0c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:12:07.421902 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.403363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35609 (local address 127.0.0.1:38508)
0928 05:12:07.415064 (+ 11701us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:07.415087 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:07.415200 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:07.416444 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:07.416452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:07.416487 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:07.416687 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.416696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.418322 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.418327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.419903 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.419914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.421176 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.421182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.421391 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.421430 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 05:12:07.421496 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 05:12:07.421623 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11568,"thread_start_us":13902,"threads_started":1}
I20260928 05:12:07.422680 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcfcae5fcc34b06b10f28806910de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:07.422966 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.403719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.0.97.3:35609)
0928 05:12:07.415819 (+ 12100us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.415830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.415866 (+ 36us) server_negotiation.cc:407] Connection header received
0928 05:12:07.415962 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.415969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.416233 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.416422 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:07.416882 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.418184 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.420045 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.420981 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.422160 (+ 1179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.422389 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.422457 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.422640 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11929,"thread_start_us":11762,"threads_started":1}
I20260928 05:12:07.424721 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.401696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33969 (local address 127.0.0.1:54362)
0928 05:12:07.411058 (+ 9362us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:07.411082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:07.411216 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:07.416693 (+ 5477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:07.416698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:07.416716 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:07.417127 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.417139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.419127 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.419132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.421134 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.421147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.422980 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.422989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.424102 (+ 1113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.424132 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:12:07.424231 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 05:12:07.424352 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9233,"thread_start_us":9110,"threads_started":1}
I20260928 05:12:07.425735 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.402451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.0.97.1:33969)
0928 05:12:07.416161 (+ 13710us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.416169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.416198 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:12:07.416281 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.416288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.416516 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.416720 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:07.417308 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.418969 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.421304 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.422774 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.422894 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.423127 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.425272 (+ 2145us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.425435 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13561,"thread_start_us":13412,"threads_started":1}
I20260928 05:12:07.426759 11133 tablet_service.cc:1467] Processing CreateTablet for tablet adcfcae5fcc34b06b10f28806910de10 (DEFAULT_TABLE table=test-table [id=ceb0bfe71bea41fa9b430ed36e7a0c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:12:07.428188 11133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcfcae5fcc34b06b10f28806910de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:07.429140 10866 tablet_service.cc:1467] Processing CreateTablet for tablet adcfcae5fcc34b06b10f28806910de10 (DEFAULT_TABLE table=test-table [id=ceb0bfe71bea41fa9b430ed36e7a0c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:12:07.430461 10866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcfcae5fcc34b06b10f28806910de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:07.434327 11216 tablet_bootstrap.cc:492] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Bootstrap starting.
I20260928 05:12:07.437322 11216 tablet_bootstrap.cc:654] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:07.438584 11216 log.cc:824] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:07.441570 11216 tablet_bootstrap.cc:492] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: No bootstrap required, opened a new log
I20260928 05:12:07.441866 11216 ts_tablet_manager.cc:1397] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 05:12:07.442456 11218 tablet_bootstrap.cc:492] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Bootstrap starting.
I20260928 05:12:07.444675 11219 tablet_bootstrap.cc:492] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Bootstrap starting.
I20260928 05:12:07.445976 11218 tablet_bootstrap.cc:654] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:07.447201 11218 log.cc:824] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:07.447340 11219 tablet_bootstrap.cc:654] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:07.448549 11219 log.cc:824] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:07.450080 11218 tablet_bootstrap.cc:492] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: No bootstrap required, opened a new log
I20260928 05:12:07.450316 11218 ts_tablet_manager.cc:1397] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 05:12:07.451058 11219 tablet_bootstrap.cc:492] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: No bootstrap required, opened a new log
I20260928 05:12:07.451337 11219 ts_tablet_manager.cc:1397] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260928 05:12:07.452857 11216 raft_consensus.cc:348] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.453270 11216 raft_consensus.cc:374] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:07.453401 11216 raft_consensus.cc:729] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9027884ce104a19a942a0e301184937, State: Initialized, Role: FOLLOWER
I20260928 05:12:07.454000 11216 consensus_queue.cc:260] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.455199 11065 heartbeater.cc:503] Master 127.0.97.62:45279 was elected leader, sending a full tablet report...
I20260928 05:12:07.455878 11216 ts_tablet_manager.cc:1428] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260928 05:12:07.460785 11218 raft_consensus.cc:348] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.461239 11218 raft_consensus.cc:374] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:07.461339 11218 raft_consensus.cc:729] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4823efc7ff084243816ee5ead34edb95, State: Initialized, Role: FOLLOWER
I20260928 05:12:07.461943 11218 consensus_queue.cc:260] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.462164 11219 raft_consensus.cc:348] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.462663 11219 raft_consensus.cc:374] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:07.462765 11219 raft_consensus.cc:729] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400b3681712c471d94b41fcf3bf27be3, State: Initialized, Role: FOLLOWER
I20260928 05:12:07.463130 11198 heartbeater.cc:503] Master 127.0.97.62:45279 was elected leader, sending a full tablet report...
I20260928 05:12:07.463338 11219 consensus_queue.cc:260] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.463790 11218 ts_tablet_manager.cc:1428] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260928 05:12:07.465341 11219 ts_tablet_manager.cc:1428] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260928 05:12:07.479334 11223 raft_consensus.cc:482] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:12:07.479588 11223 raft_consensus.cc:504] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.481113 11223 leader_election.cc:288] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9027884ce104a19a942a0e301184937 (127.0.97.2:33215), 400b3681712c471d94b41fcf3bf27be3 (127.0.97.1:33969)
I20260928 05:12:07.486809 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.481483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55501 (local address 127.0.97.2:33215)
0928 05:12:07.482161 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.482169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.482190 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:12:07.482264 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.482270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.482354 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.482456 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:07.482927 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.483608 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.485185 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.485929 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.486072 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.486158 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.486417 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.486564 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260928 05:12:07.486814 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.481486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33215 (local address 127.0.97.3:55501)
0928 05:12:07.481792 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:07.481813 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:07.481938 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:07.482581 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:07.482588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:07.482628 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:07.482785 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.482793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.483741 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.483749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.485035 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.485049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.486079 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.486084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.486266 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.486289 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:12:07.486393 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 05:12:07.486514 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":46}
I20260928 05:12:07.487762 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.481725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33969 (local address 127.0.97.3:52985)
0928 05:12:07.482822 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:07.482836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:07.482967 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:07.483271 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:07.483275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:07.483299 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:07.483601 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.483609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.484578 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.484583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.485909 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:07.485919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.487117 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.487123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:07.487297 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.487330 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:12:07.487417 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 05:12:07.487499 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":20,"thread_start_us":738,"threads_started":1}
I20260928 05:12:07.487835 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.481616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52985 (local address 127.0.97.1:33969)
0928 05:12:07.481821 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.481829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.482917 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 05:12:07.483095 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.483106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.483159 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.483246 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:07.483750 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.484439 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.486056 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.486893 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.487040 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.487122 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.487477 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.487602 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:12:07.487867 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcfcae5fcc34b06b10f28806910de10" candidate_uuid: "4823efc7ff084243816ee5ead34edb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9027884ce104a19a942a0e301184937" is_pre_election: true
I20260928 05:12:07.488420 11020 raft_consensus.cc:2454] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4823efc7ff084243816ee5ead34edb95 in term 0.
I20260928 05:12:07.488710 10886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcfcae5fcc34b06b10f28806910de10" candidate_uuid: "4823efc7ff084243816ee5ead34edb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400b3681712c471d94b41fcf3bf27be3" is_pre_election: true
I20260928 05:12:07.489117 11089 leader_election.cc:302] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4823efc7ff084243816ee5ead34edb95, d9027884ce104a19a942a0e301184937; no voters:
I20260928 05:12:07.489218 10886 raft_consensus.cc:2454] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4823efc7ff084243816ee5ead34edb95 in term 0.
I20260928 05:12:07.489475 11223 raft_consensus.cc:2781] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:12:07.489609 11223 raft_consensus.cc:482] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:12:07.489758 11223 raft_consensus.cc:3037] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:07.491994 11223 raft_consensus.cc:504] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.492586 11223 leader_election.cc:288] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [CANDIDATE]: Term 1 election: Requested vote from peers d9027884ce104a19a942a0e301184937 (127.0.97.2:33215), 400b3681712c471d94b41fcf3bf27be3 (127.0.97.1:33969)
I20260928 05:12:07.492918 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcfcae5fcc34b06b10f28806910de10" candidate_uuid: "4823efc7ff084243816ee5ead34edb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9027884ce104a19a942a0e301184937"
I20260928 05:12:07.493100 11020 raft_consensus.cc:3037] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:07.493027 10886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcfcae5fcc34b06b10f28806910de10" candidate_uuid: "4823efc7ff084243816ee5ead34edb95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400b3681712c471d94b41fcf3bf27be3"
I20260928 05:12:07.493270 10886 raft_consensus.cc:3037] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:07.495810 11020 raft_consensus.cc:2454] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4823efc7ff084243816ee5ead34edb95 in term 1.
I20260928 05:12:07.496049 10886 raft_consensus.cc:2454] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4823efc7ff084243816ee5ead34edb95 in term 1.
I20260928 05:12:07.496236 11089 leader_election.cc:302] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4823efc7ff084243816ee5ead34edb95, d9027884ce104a19a942a0e301184937; no voters:
I20260928 05:12:07.496687 11223 raft_consensus.cc:2781] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:12:07.497218 11223 raft_consensus.cc:686] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 1 LEADER]: Becoming Leader. State: Replica: 4823efc7ff084243816ee5ead34edb95, State: Running, Role: LEADER
I20260928 05:12:07.497900 11223 consensus_queue.cc:237] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260928 05:12:07.502997 10741 catalog_manager.cc:5495] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4823efc7ff084243816ee5ead34edb95 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4823efc7ff084243816ee5ead34edb95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4823efc7ff084243816ee5ead34edb95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } health_report { overall_health: HEALTHY } } }
I20260928 05:12:07.536039 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.530826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.0.97.3:35609)
0928 05:12:07.531028 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.531035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.531130 (+ 95us) server_negotiation.cc:407] Connection header received
0928 05:12:07.531374 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.531378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.531456 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.531607 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:12:07.531986 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.532696 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.534000 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.534378 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.534509 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.535018 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:12:07.535491 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:12:07.535660 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.535709 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.535783 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:12:07.543864 10886 raft_consensus.cc:1261] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 1 FOLLOWER]: Refusing update from remote peer 4823efc7ff084243816ee5ead34edb95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:12:07.544169 11020 raft_consensus.cc:1261] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 1 FOLLOWER]: Refusing update from remote peer 4823efc7ff084243816ee5ead34edb95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:12:07.544859 11223 consensus_queue.cc:1035] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "400b3681712c471d94b41fcf3bf27be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:12:07.545182 11226 consensus_queue.cc:1035] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9027884ce104a19a942a0e301184937" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:12:07.563555 11230 mvcc.cc:205] Tried to move back new op lower bound from 7334184253610307584 to 7334184253436243968. Current Snapshot: MvccSnapshot[applied={T|T < 7334184253610307584}]
I20260928 05:12:07.570894 10741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:12:07.571167 10741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:12:07.574788 10741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69271b846244272ad3aec3ad75c9ad7: Sending DeleteTablet for 3 replicas of tablet adcfcae5fcc34b06b10f28806910de10
I20260928 05:12:07.575924 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet adcfcae5fcc34b06b10f28806910de10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:12:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53454
I20260928 05:12:07.576009 10866 tablet_service.cc:1514] Processing DeleteTablet for tablet adcfcae5fcc34b06b10f28806910de10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:12:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54362
I20260928 05:12:07.576231 11133 tablet_service.cc:1514] Processing DeleteTablet for tablet adcfcae5fcc34b06b10f28806910de10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:12:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260928 05:12:07.583173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10803
I20260928 05:12:07.589112 11239 tablet_replica.cc:330] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: stopping tablet replica
I20260928 05:12:07.589578 11239 raft_consensus.cc:2229] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:12:07.590101 11239 raft_consensus.cc:2258] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:12:07.590929 11238 tablet_replica.cc:330] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: stopping tablet replica
I20260928 05:12:07.590929 11240 tablet_replica.cc:330] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: stopping tablet replica
I20260928 05:12:07.591326 11238 raft_consensus.cc:2229] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:12:07.591328 11240 raft_consensus.cc:2229] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:12:07.591796 11238 raft_consensus.cc:2258] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:12:07.592013 11240 raft_consensus.cc:2258] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:12:07.593974 11240 ts_tablet_manager.cc:1905] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:12:07.593976 11238 ts_tablet_manager.cc:1905] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:12:07 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:12:07.600765 11238 ts_tablet_manager.cc:1918] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:12:07.600765 11240 ts_tablet_manager.cc:1918] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:12:07.600926 11238 log.cc:1196] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/adcfcae5fcc34b06b10f28806910de10
I20260928 05:12:07.600939 11240 log.cc:1196] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/adcfcae5fcc34b06b10f28806910de10
I20260928 05:12:07.601449 11238 ts_tablet_manager.cc:1939] T adcfcae5fcc34b06b10f28806910de10 P d9027884ce104a19a942a0e301184937: Deleting consensus metadata
I20260928 05:12:07.601449 11240 ts_tablet_manager.cc:1939] T adcfcae5fcc34b06b10f28806910de10 P 4823efc7ff084243816ee5ead34edb95: Deleting consensus metadata
I20260928 05:12:07.603904 11239 ts_tablet_manager.cc:1905] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:12:07.605017 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.584927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.0.97.1:33969)
0928 05:12:07.585161 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.585169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.585195 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:12:07.585272 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.585280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.585423 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.585567 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:07.592084 (+ 6517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.593133 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.594632 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.595035 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.595171 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.599070 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:07.599081 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:07.599094 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:07.599126 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:07.602042 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.602759 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.602766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.602768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.602886 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.603280 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.603284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.603287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.603660 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:07.603981 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.604309 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.604540 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:12:07.605715 10926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:07.606583 10729 catalog_manager.cc:4841] TS d9027884ce104a19a942a0e301184937 (127.0.97.2:33215): tablet adcfcae5fcc34b06b10f28806910de10 (table test-table [id=ceb0bfe71bea41fa9b430ed36e7a0c5d]) successfully deleted
I20260928 05:12:07.606917 10729 catalog_manager.cc:4841] TS 4823efc7ff084243816ee5ead34edb95 (127.0.97.3:35609): tablet adcfcae5fcc34b06b10f28806910de10 (table test-table [id=ceb0bfe71bea41fa9b430ed36e7a0c5d]) successfully deleted
I20260928 05:12:07.700873 11239 ts_tablet_manager.cc:1918] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:12:07.701164 11239 log.cc:1196] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/adcfcae5fcc34b06b10f28806910de10
I20260928 05:12:07.701740 11239 ts_tablet_manager.cc:1939] T adcfcae5fcc34b06b10f28806910de10 P 400b3681712c471d94b41fcf3bf27be3: Deleting consensus metadata
I20260928 05:12:07.703728 10728 catalog_manager.cc:4841] TS 400b3681712c471d94b41fcf3bf27be3 (127.0.97.1:33969): tablet adcfcae5fcc34b06b10f28806910de10 (table test-table [id=ceb0bfe71bea41fa9b430ed36e7a0c5d]) successfully deleted
I20260928 05:12:07.706703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10803
I20260928 05:12:07.727294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10937
Sep 28 05:12:07 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:12:07.744175 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.728182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.0.97.2:33215)
0928 05:12:07.728379 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.728385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.728404 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:12:07.728461 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.728466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.728554 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.728669 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:07.729750 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.730884 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.732084 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.732725 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.732850 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.736793 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:07.736812 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:07.736834 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:07.736874 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:07.740736 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.741573 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.741683 (+ 110us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.741687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.741844 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.742383 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.742389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.742393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.742954 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:07.743254 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.743553 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.743807 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:12:07.745081 11060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:07.833839 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10937
I20260928 05:12:07.857645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 11070
Sep 28 05:12:07 dist-test-slave-6718 krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:12:07.876926 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:07.858478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.0.97.3:35609)
0928 05:12:07.858686 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:07.858691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:07.858766 (+ 75us) server_negotiation.cc:407] Connection header received
0928 05:12:07.859033 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:07.859037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:07.859126 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:07.859204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:07.860858 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.861623 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.863809 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:07.864318 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:07.864448 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:07.869852 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:07.869863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:07.869878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:07.869921 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:07.873585 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.874451 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.874460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.874464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.874675 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:07.875076 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:07.875082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:07.875085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:07.875538 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:07.875843 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:07.876261 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 05:12:07.876483 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 05:12:07.877737 11193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:07.997740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 11070
I20260928 05:12:08.016716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 10710
I20260928 05:12:08.017843 10772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:08.125696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 10710
2026-09-28T05:12:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4238 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:12:08 dist-test-slave-6718 krb5kdc[11248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:12:08 dist-test-slave-6718 krb5kdc[11248](info): set up 2 sockets
Sep 28 05:12:08 dist-test-slave-6718 krb5kdc[11248](info): commencing operation
krb5kdc: starting...
W20260928 05:12:09.215410 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.009s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:12:09 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:12:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 05:12:09.526408 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzfOJKc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/postgres -l logfile start
2026-09-28 05:12:12.041 UTC [11278] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 05:12:12.041 UTC [11278] LOG: listening on IPv4 address "127.0.97.20", port 48733
2026-09-28 05:12:12.044 UTC [11278] LOG: listening on Unix socket "/tmp/.s.PGSQL.48733"
2026-09-28 05:12:12.067 UTC [11281] LOG: database system was shut down at 2026-09-28 05:12:10 UTC
2026-09-28 05:12:12.070 UTC [11278] LOG: database system is ready to accept connections
I20260928 05:12:13.047231 388 mini_postgres.cc:96] Postgres bound to 48733
2026-09-28 05:12:13.063 UTC [11289] FATAL: database "slave" does not exist
127.0.97.20:48733 - accepting connections
I20260928 05:12:13.064757 388 mini_ranger.cc:161] Starting Ranger...
I20260928 05:12:13.088919 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 05:12:13.548362 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 05:12:13.548537 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-admin
2026-09-28 05:12:13,837 [I] DB FLAVOR :POSTGRES
2026-09-28 05:12:13,837 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:12:13,837 [I] Checking connection..
2026-09-28 05:12:13,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 05:12:14,101 [I] Checking connection passed.
2026-09-28 05:12:14,101 [I] --------- Verifying version history table ---------
2026-09-28 05:12:14,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:12:14,360 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:12:14,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:12:14,617 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:12:14,618 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 05:12:14,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 05:12:14.845 UTC [11353] WARNING: there is no transaction in progress
2026-09-28 05:12:14,851 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 05:12:14,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:12:15,107 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 05:12:15,108 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:12:15,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 05:12:15,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 05:12:15,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6718.c.gcp-upstream.internal','N') ;"
2026-09-28 05:12:15,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:12:16,103 [I] Table x_portal_user does not exist in database ranger
2026-09-28 05:12:16,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:12:16,366 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 05:12:16,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:12:16,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:12:16,847 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 05:12:16,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 05:12:17.766 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.782 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.797 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.812 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.827 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.844 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.909 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.915 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.930 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.940 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.950 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.964 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.973 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.980 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.987 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:17.994 UTC [11461] WARNING: there is no transaction in progress
2026-09-28 05:12:18,595 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 05:12:18,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 05:12:18,821 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 05:12:18,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:12:19,080 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 05:12:19,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:12:19,316 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 05:12:19,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:12:19,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:12:19,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6718.c.gcp-upstream.internal';"
2026-09-28 05:12:20,018 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 05:12:20,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 05:12:20,275 [I] DB_PATCHES have already been applied
I20260928 05:12:20.281558 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzfOJKc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 05:12:20.281662 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 05:12:20.296801 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:52559
Sep 28, 2026 5:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:12:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:12:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:12:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 05:12:21.429003 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.364s user 0.000s sys 0.039s
Sep 28, 2026 5:12:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:12:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:12:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:12:40 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:12:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 05:12:42.324105 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 05:12:42 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 05:12:42.499212 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 05:12:42.522634 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 05:12:43.341982 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 05:12:43.342125 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms
I20260928 05:12:43.342270 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48733
2026-09-28 05:12:43,750 [I] DB FLAVOR :POSTGRES
2026-09-28 05:12:43,750 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:12:43,750 [I] Checking connection
2026-09-28 05:12:43,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 05:12:43,981 [I] connection success
2026-09-28 05:12:43,981 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 05:12:43,981 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 05:12:43,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 05:12:44,210 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 05:12:44,210 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:12:44,210 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 05:12:44,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48733/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 05:12:44,469 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 05:12:44.912994 388 mini_ranger_kms.cc:310] Created kms service
I20260928 05:12:45.124612 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 05:12:45.169360 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 05:12:45 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:12:45.550439 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 05:12:45.550900 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 05:12:45.550983 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 05:12:45.568481 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11641
I20260928 05:12:45.568701 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:34455
Sep 28, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:12:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:12:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 05:12:47.887017 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.388s user 0.126s sys 0.302s
I20260928 05:12:47.887262 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 05:12:47.887462 388 mini_ranger_kms.cc:401] 127.0.97.20:34455/kms/v1/keys
Sep 28 05:12:48 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 05:12:48 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:12:48 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:12:51 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:12:51.115046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36483
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36483
--ranger_config_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34455/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:12:51.472378 11692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:51.472648 11692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:51.472739 11692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:51.480026 11692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:12:51.480134 11692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:51.480177 11692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:51.480229 11692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:12:51.480274 11692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:12:51.489862 11692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:34455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36483
--ranger_config_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:51.491058 11692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:51.492238 11692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:51.504401 11697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:51.504320 11698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:51.504766 11692 server_base.cc:1034] running on GCE node
W20260928 05:12:51.504380 11700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:51.505663 11692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:51.506978 11692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:51.508196 11692 hybrid_clock.cc:648] HybridClock initialized: now 1790572371508156 us; error 24 us; skew 500 ppm
I20260928 05:12:51.508574 11692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:51 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:51.515093 11692 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:12:51.516888 11692 webserver.cc:460] Webserver started at http://127.0.97.62:34499/ using document root <none> and password file <none>
I20260928 05:12:51.517535 11692 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:51.517632 11692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:51.517947 11692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:12:51 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:12:51.569914 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "44d5b8f2b0a74189bf07e2a07a2a593e"
format_stamp: "Formatted at 2026-09-28 05:12:51 on dist-test-slave-6718"
server_key: "80195529607d9d03123bd6bf8e830bb6"
server_key_iv: "ecf64cd725d97e5951a86ea3f0564d70"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:51.570848 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "44d5b8f2b0a74189bf07e2a07a2a593e"
format_stamp: "Formatted at 2026-09-28 05:12:51 on dist-test-slave-6718"
server_key: "80195529607d9d03123bd6bf8e830bb6"
server_key_iv: "ecf64cd725d97e5951a86ea3f0564d70"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:51.576292 11692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 05:12:51.592604 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:51.594159 11692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:12:51.594360 11692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "44d5b8f2b0a74189bf07e2a07a2a593e"
format_stamp: "Formatted at 2026-09-28 05:12:51 on dist-test-slave-6718"
server_key: "80195529607d9d03123bd6bf8e830bb6"
server_key_iv: "ecf64cd725d97e5951a86ea3f0564d70"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:51.594655 11692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:51.606328 11692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:51.610265 11692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:51.610615 11692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:51.631657 11692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36483
I20260928 05:12:51.631693 11758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36483 every 8 connection(s)
I20260928 05:12:51.633056 11692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:12:51.636803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 11692
I20260928 05:12:51.637001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:12:51.637403 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:51.637744 388 external_mini_cluster.cc:1428] Setting key aa337f034a57b7293811fc95a4a9219c
I20260928 05:12:51.655596 11759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e: Bootstrap starting.
I20260928 05:12:51.659036 11759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:51.660189 11759 log.cc:824] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:51.663766 11759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e: No bootstrap required, opened a new log
Sep 28 05:12:51 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:51.674171 11759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36483 } }
I20260928 05:12:51.674623 11759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:51.674746 11759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d5b8f2b0a74189bf07e2a07a2a593e, State: Initialized, Role: FOLLOWER
I20260928 05:12:51.675521 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:51.639817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.0.97.62:36483)
0928 05:12:51.654519 (+ 14702us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:51.654558 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:51.654608 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:12:51.656975 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:51.657006 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:51.657519 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:51.657967 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:51.659600 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:51.661618 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:51.663360 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:51.664174 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:51.664373 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:51.668175 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:51.668183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:51.668197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:51.668229 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:51.671268 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:51.671950 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:51.671956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:51.671962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:51.672120 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:51.672498 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:51.672507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:51.672509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:51.673035 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:51.673343 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:51.673789 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 05:12:51.674214 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14461,"thread_start_us":14671,"threads_started":1}
I20260928 05:12:51.675424 11759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36483 } }
I20260928 05:12:51.675674 11759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:12:51.675845 11759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:12:51.676074 11759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:51.679275 11759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36483 } }
I20260928 05:12:51.680346 11759 leader_election.cc:302] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44d5b8f2b0a74189bf07e2a07a2a593e; no voters:
I20260928 05:12:51.680992 11759 leader_election.cc:288] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:12:51.681116 11764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:12:51.682814 11764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [term 1 LEADER]: Becoming Leader. State: Replica: 44d5b8f2b0a74189bf07e2a07a2a593e, State: Running, Role: LEADER
I20260928 05:12:51.683290 11759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:12:51.683420 11764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36483 } }
I20260928 05:12:51.686256 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36483 } } }
I20260928 05:12:51.686268 11766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44d5b8f2b0a74189bf07e2a07a2a593e. Latest consensus state: current_term: 1 leader_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d5b8f2b0a74189bf07e2a07a2a593e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36483 } } }
I20260928 05:12:51.686798 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [sys.catalog]: This master's current role is: LEADER
I20260928 05:12:51.686832 11766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e [sys.catalog]: This master's current role is: LEADER
I20260928 05:12:51.695444 11770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:12:51.702356 11770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:12:51.727514 11770 catalog_manager.cc:1347] Generated new cluster ID: 3b32733f0c6640bb8aabb9a7d8e2ebec
I20260928 05:12:51.727630 11770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:12:51.751127 11770 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:12:51.753021 11770 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:12:51.771061 11770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e: Generated new TSK 0
I20260928 05:12:51.772226 11770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:12:51.777184 11759 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:12:52.566501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36483
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:12:52 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:12:53.003918 11804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:53.004391 11804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:53.004583 11804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:53.016630 11804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:53.016791 11804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:53.016975 11804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:12:53.033033 11804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:34455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:53.034865 11804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:53.036443 11804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:53.050118 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:53.050381 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:53.050565 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:53.051301 11804 server_base.cc:1034] running on GCE node
I20260928 05:12:53.051903 11804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:53.052906 11804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:53.054126 11804 hybrid_clock.cc:648] HybridClock initialized: now 1790572373054041 us; error 77 us; skew 500 ppm
I20260928 05:12:53.054437 11804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:53 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:53.057569 11804 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:12:53.059479 11804 webserver.cc:460] Webserver started at http://127.0.97.1:34035/ using document root <none> and password file <none>
I20260928 05:12:53.060146 11804 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:53.060280 11804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:53.060598 11804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:12:53 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:12:53.084031 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "d2f06ee8bacc489d822a4d269bb47609"
format_stamp: "Formatted at 2026-09-28 05:12:53 on dist-test-slave-6718"
server_key: "38fca561d8e7f39169f308f605950531"
server_key_iv: "a4fef9ef6cc88ebb6cf6daf30602e897"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:53.085196 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d2f06ee8bacc489d822a4d269bb47609"
format_stamp: "Formatted at 2026-09-28 05:12:53 on dist-test-slave-6718"
server_key: "38fca561d8e7f39169f308f605950531"
server_key_iv: "a4fef9ef6cc88ebb6cf6daf30602e897"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:53.104923 11804 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.007s sys 0.012s
I20260928 05:12:53.139725 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:53.146590 11804 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.008s
I20260928 05:12:53.146853 11804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "d2f06ee8bacc489d822a4d269bb47609"
format_stamp: "Formatted at 2026-09-28 05:12:53 on dist-test-slave-6718"
server_key: "38fca561d8e7f39169f308f605950531"
server_key_iv: "a4fef9ef6cc88ebb6cf6daf30602e897"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:53.147193 11804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:53.222656 11804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:53.227910 11804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:53.228279 11804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:53.239431 11804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:53.247767 11804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:53.247946 11804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 05:12:53.248150 11804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:53.248282 11804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:12:53 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:12:53 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:12:53.918469 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39433 every 8 connection(s)
I20260928 05:12:53.918828 11804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39433
I20260928 05:12:53.920782 11804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:12:53.930603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 11804
I20260928 05:12:53.930801 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:12:53.931617 388 external_mini_cluster.cc:1428] Setting key 12d68f4bf2cdd9bb43d922dc2fbf2f1b
Sep 28 05:12:53 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:12:53.981952 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:53.934232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56923 (local address 127.0.97.62:36483)
0928 05:12:53.952920 (+ 18688us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:53.952928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:53.952957 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:12:53.953044 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:53.953052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:53.953483 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:53.954002 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:53.956533 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:53.957761 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:53.959810 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:53.960395 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:53.960516 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:53.965133 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:53.965140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:53.965144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:53.965193 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:53.977472 (+ 12279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:53.978505 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:53.978511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:53.978515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:53.978690 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:53.979256 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:53.979264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:53.979267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:53.979648 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:53.979797 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:53.980922 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0928 05:12:53.981162 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18504,"thread_start_us":18359,"threads_started":1}
I20260928 05:12:53.983394 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:53.935033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36483 (local address 127.0.97.1:56923)
0928 05:12:53.943216 (+ 8183us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:53.943278 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:53.946056 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:53.954068 (+ 8012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:53.954084 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:53.955306 (+ 1222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:53.956270 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:53.956295 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:53.957965 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:53.957973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:53.959628 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:53.959644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:53.960838 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:53.960846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:53.961120 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:53.961936 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:12:53.961971 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:53.964905 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:53.977657 (+ 12752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:53.977668 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:53.977689 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:53.978339 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:53.978823 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:53.978828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:53.978832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:53.979105 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:53.979808 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:53.979819 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:53.980446 (+ 627us) client_negotiation.cc:769] Sending connection context
0928 05:12:53.980823 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 05:12:53.981860 (+ 1037us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7800,"thread_start_us":7535,"threads_started":1}
I20260928 05:12:53.985054 11938 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36483
I20260928 05:12:53.985416 11938 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:53.986280 11938 heartbeater.cc:511] Master 127.0.97.62:36483 requested a full tablet report, sending...
I20260928 05:12:53.988870 11724 ts_manager.cc:194] Registered new tserver with Master: d2f06ee8bacc489d822a4d269bb47609 (127.0.97.1:39433)
I20260928 05:12:53.991146 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:12:54.021067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36483
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:12:54.493868 11945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:54.494254 11945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:54.494387 11945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:54.505331 11945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:54.505492 11945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:54.505725 11945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:12:54.521477 11945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:34455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:54.523126 11945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:54.524940 11945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:54.541599 11957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:54.542308 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:54.547603 11945 server_base.cc:1034] running on GCE node
W20260928 05:12:54.548514 11960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:54.549206 11945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:54.550248 11945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:54.551508 11945 hybrid_clock.cc:648] HybridClock initialized: now 1790572374551419 us; error 83 us; skew 500 ppm
I20260928 05:12:54.551893 11945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:54 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:54.555866 11945 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:12:54.557819 11945 webserver.cc:460] Webserver started at http://127.0.97.2:37911/ using document root <none> and password file <none>
I20260928 05:12:54.558637 11945 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:54.558779 11945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:54.559134 11945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:12:54 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:12:54.577039 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "091c5a9e0f0640849efeb0c13dea352b"
format_stamp: "Formatted at 2026-09-28 05:12:54 on dist-test-slave-6718"
server_key: "44adb76525e0d64b28884068afcc28af"
server_key_iv: "a1abdf9b52e4642af37ed6e80d7ed5f3"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:54.584353 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "091c5a9e0f0640849efeb0c13dea352b"
format_stamp: "Formatted at 2026-09-28 05:12:54 on dist-test-slave-6718"
server_key: "44adb76525e0d64b28884068afcc28af"
server_key_iv: "a1abdf9b52e4642af37ed6e80d7ed5f3"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:54.604646 11945 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.011s sys 0.007s
I20260928 05:12:54.638402 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:54.640772 11945 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260928 05:12:54.641047 11945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "091c5a9e0f0640849efeb0c13dea352b"
format_stamp: "Formatted at 2026-09-28 05:12:54 on dist-test-slave-6718"
server_key: "44adb76525e0d64b28884068afcc28af"
server_key_iv: "a1abdf9b52e4642af37ed6e80d7ed5f3"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:54.641361 11945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:54.681608 11945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:54.687171 11945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:54.688002 11945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:54.690277 11945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:54.692147 11945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:54.692278 11945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 05:12:54.692422 11945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:54.692507 11945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:54.741884 11945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40223
I20260928 05:12:54.741930 12079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40223 every 8 connection(s)
I20260928 05:12:54.743223 11945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:12:54.750166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 11945
I20260928 05:12:54.750770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:12:54.751312 388 external_mini_cluster.cc:1428] Setting key 6e879d4f0fcafc6102a26a4285e60285
Sep 28 05:12:54 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:54.766315 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:54.745860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45095 (local address 127.0.97.62:36483)
0928 05:12:54.746351 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:54.746357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:54.747568 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 05:12:54.749442 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:54.749446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:54.749511 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:54.749606 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:54.751980 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:54.752830 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:54.755327 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:54.755740 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:54.755844 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:54.760129 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:54.760138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:54.760146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:54.760199 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:54.763130 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:54.764029 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:54.764033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:54.764035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:54.764142 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:54.764611 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:54.764614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:54.764616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:54.764908 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:54.765013 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:54.765867 (+ 854us) server_negotiation.cc:299] Negotiation successful
0928 05:12:54.766024 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260928 05:12:54.767741 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:54.746506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36483 (local address 127.0.97.2:45095)
0928 05:12:54.747406 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:54.747454 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:54.749201 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:54.749891 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:54.749902 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:54.750967 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:54.751759 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:54.751784 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:54.752979 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:54.752985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:54.755171 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:54.755201 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:54.755983 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:54.755993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:54.756340 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:54.757106 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:12:54.757136 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:54.759882 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:54.763273 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:54.763284 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:54.763304 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:54.763913 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:54.764245 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:54.764252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:54.764258 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:54.764499 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:54.765038 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:54.765049 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:54.765466 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 05:12:54.765762 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 05:12:54.766298 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":335,"threads_started":1}
I20260928 05:12:54.769194 12080 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36483
I20260928 05:12:54.769484 12080 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:54.770165 12080 heartbeater.cc:511] Master 127.0.97.62:36483 requested a full tablet report, sending...
I20260928 05:12:54.771699 11724 ts_manager.cc:194] Registered new tserver with Master: 091c5a9e0f0640849efeb0c13dea352b (127.0.97.2:40223)
I20260928 05:12:54.772469 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45095
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:12:54.804065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:34455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36483
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:12:54.995009 11938 heartbeater.cc:503] Master 127.0.97.62:36483 was elected leader, sending a full tablet report...
W20260928 05:12:55.131490 12086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:12:55.131763 12086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:12:55.131851 12086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:12:55.139792 12086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:12:55.139892 12086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:12:55.140000 12086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:12:55.149674 12086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:34455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:12:55.150913 12086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:12:55.152120 12086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:12:55.165288 12091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:55.165263 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:12:55.165261 12092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:12:55.165974 12086 server_base.cc:1034] running on GCE node
I20260928 05:12:55.166620 12086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:12:55.167562 12086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:12:55.168789 12086 hybrid_clock.cc:648] HybridClock initialized: now 1790572375168736 us; error 47 us; skew 500 ppm
I20260928 05:12:55.169107 12086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:12:55 dist-test-slave-6718 krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:12:55.172545 12086 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:12:55.173910 12086 webserver.cc:460] Webserver started at http://127.0.97.3:42667/ using document root <none> and password file <none>
I20260928 05:12:55.174499 12086 fs_manager.cc:359] Metadata directory not provided
I20260928 05:12:55.174665 12086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:12:55.174924 12086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:12:55 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:12:55.191773 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "5f1b15b2451a4f6989b4ae0a9be405a6"
format_stamp: "Formatted at 2026-09-28 05:12:55 on dist-test-slave-6718"
server_key: "bacf947a62b1529a730263a8d027cf46"
server_key_iv: "e74ca103dd7bdc538e243d4799d64718"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:55.192806 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5f1b15b2451a4f6989b4ae0a9be405a6"
format_stamp: "Formatted at 2026-09-28 05:12:55 on dist-test-slave-6718"
server_key: "bacf947a62b1529a730263a8d027cf46"
server_key_iv: "e74ca103dd7bdc538e243d4799d64718"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:55.199556 12086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 05:12:55.215548 12101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:55.217159 12086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:12:55.217338 12086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "5f1b15b2451a4f6989b4ae0a9be405a6"
format_stamp: "Formatted at 2026-09-28 05:12:55 on dist-test-slave-6718"
server_key: "bacf947a62b1529a730263a8d027cf46"
server_key_iv: "e74ca103dd7bdc538e243d4799d64718"
server_key_version: "kuduclusterkey@0"
I20260928 05:12:55.218003 12086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:12:55.283985 12086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:12:55.287885 12086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:12:55.288177 12086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:12:55.289137 12086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:12:55.290434 12086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:12:55.290504 12086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:55.290661 12086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:12:55.290719 12086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:12:55.325881 12086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37567
I20260928 05:12:55.325901 12213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37567 every 8 connection(s)
I20260928 05:12:55.327224 12086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:12:55.336522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 12086
I20260928 05:12:55.336694 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:12:55.337255 388 external_mini_cluster.cc:1428] Setting key 90e5be50489b78b059284982fa0de56c
Sep 28 05:12:55 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:12:55.347618 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.329615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54831 (local address 127.0.97.62:36483)
0928 05:12:55.330040 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.330046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.331152 (+ 1106us) server_negotiation.cc:407] Connection header received
0928 05:12:55.332803 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.332808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.332889 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.332985 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:55.334308 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.335070 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.336811 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.337455 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.337592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.341364 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:55.341370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:55.341377 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:55.341416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:55.344536 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.345337 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.345342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.345344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.345456 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.345896 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.345904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.345907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.346165 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:55.346339 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.347043 (+ 704us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.347245 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":210,"threads_started":1}
I20260928 05:12:55.348553 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.330254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36483 (local address 127.0.97.3:54831)
0928 05:12:55.330993 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:55.331029 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:55.332552 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:55.333207 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:55.333215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:55.333612 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:12:55.334155 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.334171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.335190 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.335194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.336628 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.336645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.337650 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.337655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.337936 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.338709 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:12:55.338736 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:12:55.341160 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:12:55.344689 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:55.344698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:55.344716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:55.345201 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:55.345567 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:12:55.345571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:12:55.345575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:12:55.345774 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:12:55.346308 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:12:55.346316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:12:55.346695 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 05:12:55.346973 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 05:12:55.347460 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":333,"threads_started":1}
I20260928 05:12:55.349879 12214 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36483
I20260928 05:12:55.350176 12214 heartbeater.cc:464] Registering TS with master...
I20260928 05:12:55.350975 12214 heartbeater.cc:511] Master 127.0.97.62:36483 requested a full tablet report, sending...
I20260928 05:12:55.352588 11723 ts_manager.cc:194] Registered new tserver with Master: 5f1b15b2451a4f6989b4ae0a9be405a6 (127.0.97.3:37567)
I20260928 05:12:55.353502 11723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54831
I20260928 05:12:55.364960 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:12:55.381698 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.369489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.0.97.62:36483)
0928 05:12:55.369711 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.369717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.370119 (+ 402us) server_negotiation.cc:407] Connection header received
0928 05:12:55.370244 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.370248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.370321 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.370408 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:55.371795 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.372532 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.373829 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.374241 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.374332 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.376880 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:55.376885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:55.376888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:55.376918 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:55.379498 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.380169 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.380173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.380176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.380308 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.380616 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.380619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.380621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.380861 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:55.381027 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.381328 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.381484 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 05:12:55.385767 11723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:12:55.392989 11723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:12:55.417074 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.405340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.0.97.2:40223)
0928 05:12:55.406810 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.406822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.407461 (+ 639us) server_negotiation.cc:407] Connection header received
0928 05:12:55.407875 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.407883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.408134 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.408300 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:55.409280 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.411310 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.414072 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.415541 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.415695 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.415903 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.416584 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.416756 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":1196,"threads_started":1}
I20260928 05:12:55.418098 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.405116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40223 (local address 127.0.0.1:59462)
0928 05:12:55.407295 (+ 2179us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:55.407334 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:55.407629 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:55.408595 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:55.408601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:55.408634 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:55.409098 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.409113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.411487 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.411492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.413893 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.413911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.416094 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.416101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.416286 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.416351 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 05:12:55.417559 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0928 05:12:55.417739 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"thread_start_us":1761,"threads_started":1}
I20260928 05:12:55.419664 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.405867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39433 (local address 127.0.0.1:41320)
0928 05:12:55.407762 (+ 1895us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:55.407780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:55.407879 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:55.409749 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:55.409754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:55.409778 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:55.410148 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.410159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.413591 (+ 3432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.413596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.415700 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.415713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.418850 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.418858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.419078 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.419115 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 05:12:55.419248 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 05:12:55.419369 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"thread_start_us":1588,"threads_started":1}
I20260928 05:12:55.420380 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.406145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.97.1:39433)
0928 05:12:55.407307 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.407315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.407995 (+ 680us) server_negotiation.cc:407] Connection header received
0928 05:12:55.408089 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.408096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.408299 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.409502 (+ 1203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:55.410356 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.413424 (+ 3068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.415863 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.418394 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.418528 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.418780 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.419320 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.419556 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":1790,"threads_started":1}
I20260928 05:12:55.424124 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 66e30a99b9264155b74bb758630548d6 (DEFAULT_TABLE table=test-table [id=266ebf3cb552493789bb4dec88e2f6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:12:55.424124 11872 tablet_service.cc:1467] Processing CreateTablet for tablet 66e30a99b9264155b74bb758630548d6 (DEFAULT_TABLE table=test-table [id=266ebf3cb552493789bb4dec88e2f6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:12:55.425576 11872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e30a99b9264155b74bb758630548d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:55.425577 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e30a99b9264155b74bb758630548d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:55.425930 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.407845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37567 (local address 127.0.0.1:38030)
0928 05:12:55.418524 (+ 10679us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:55.418541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:55.418645 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:55.420071 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:55.420078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:55.420111 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:55.420317 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.420333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.422351 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.422356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.423567 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.423579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.425224 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.425230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.425397 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.425429 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:12:55.425526 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:12:55.425663 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10568,"thread_start_us":12349,"threads_started":1}
I20260928 05:12:55.426066 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.408213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.0.97.3:37567)
0928 05:12:55.419271 (+ 11058us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.419279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.419307 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:12:55.419390 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.419397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.419654 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.419849 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:55.420461 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.422214 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.423709 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.425061 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.425214 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.425423 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.425581 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.425735 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10882,"thread_start_us":10713,"threads_started":1}
I20260928 05:12:55.429302 12149 tablet_service.cc:1467] Processing CreateTablet for tablet 66e30a99b9264155b74bb758630548d6 (DEFAULT_TABLE table=test-table [id=266ebf3cb552493789bb4dec88e2f6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:12:55.430460 12149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e30a99b9264155b74bb758630548d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:12:55.444738 12233 tablet_bootstrap.cc:492] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Bootstrap starting.
I20260928 05:12:55.447479 12234 tablet_bootstrap.cc:492] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Bootstrap starting.
I20260928 05:12:55.449617 12233 tablet_bootstrap.cc:654] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:55.450100 12234 tablet_bootstrap.cc:654] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:55.451256 12233 log.cc:824] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:55.451879 12235 tablet_bootstrap.cc:492] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Bootstrap starting.
I20260928 05:12:55.453197 12234 log.cc:824] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:55.454846 12233 tablet_bootstrap.cc:492] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: No bootstrap required, opened a new log
I20260928 05:12:55.455134 12233 ts_tablet_manager.cc:1397] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 05:12:55.455195 12235 tablet_bootstrap.cc:654] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Neither blocks nor log segments found. Creating new log.
I20260928 05:12:55.456073 12235 log.cc:824] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Log is configured to *not* fsync() on all Append() calls
I20260928 05:12:55.456914 12234 tablet_bootstrap.cc:492] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: No bootstrap required, opened a new log
I20260928 05:12:55.457163 12234 ts_tablet_manager.cc:1397] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 05:12:55.458449 12235 tablet_bootstrap.cc:492] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: No bootstrap required, opened a new log
I20260928 05:12:55.458716 12235 ts_tablet_manager.cc:1397] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260928 05:12:55.465265 12235 raft_consensus.cc:348] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.465649 12235 raft_consensus.cc:374] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:55.465807 12235 raft_consensus.cc:729] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1b15b2451a4f6989b4ae0a9be405a6, State: Initialized, Role: FOLLOWER
I20260928 05:12:55.466331 12233 raft_consensus.cc:348] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.466495 12235 consensus_queue.cc:260] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.466814 12233 raft_consensus.cc:374] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:55.466993 12233 raft_consensus.cc:729] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f06ee8bacc489d822a4d269bb47609, State: Initialized, Role: FOLLOWER
I20260928 05:12:55.467487 12234 raft_consensus.cc:348] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.467617 12233 consensus_queue.cc:260] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.467919 12234 raft_consensus.cc:374] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:12:55.468063 12234 raft_consensus.cc:729] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091c5a9e0f0640849efeb0c13dea352b, State: Initialized, Role: FOLLOWER
I20260928 05:12:55.468724 12214 heartbeater.cc:503] Master 127.0.97.62:36483 was elected leader, sending a full tablet report...
I20260928 05:12:55.469336 12235 ts_tablet_manager.cc:1428] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260928 05:12:55.469686 12233 ts_tablet_manager.cc:1428] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 05:12:55.470280 12234 consensus_queue.cc:260] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.471359 12080 heartbeater.cc:503] Master 127.0.97.62:36483 was elected leader, sending a full tablet report...
I20260928 05:12:55.473798 12234 ts_tablet_manager.cc:1428] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
W20260928 05:12:55.495352 12081 tablet.cc:2369] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:12:55.578812 12215 tablet.cc:2369] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:12:55.696321 12241 raft_consensus.cc:482] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:12:55.696669 12241 raft_consensus.cc:504] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.698638 12241 leader_election.cc:288] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1b15b2451a4f6989b4ae0a9be405a6 (127.0.97.3:37567), d2f06ee8bacc489d822a4d269bb47609 (127.0.97.1:39433)
I20260928 05:12:55.705796 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.699020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43719 (local address 127.0.97.3:37567)
0928 05:12:55.699242 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.699249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.700074 (+ 825us) server_negotiation.cc:407] Connection header received
0928 05:12:55.700245 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.700249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.700301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.700407 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:55.701460 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.702130 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.704092 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.704858 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.704953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.705080 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.705415 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.705545 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:12:55.705806 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.699424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48875 (local address 127.0.97.1:39433)
0928 05:12:55.699619 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.699626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.699968 (+ 342us) server_negotiation.cc:407] Connection header received
0928 05:12:55.700170 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.700178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.700263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.700329 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:12:55.701362 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.702102 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.704140 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.704877 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.705035 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.705142 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.705416 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.705548 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:12:55.706004 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.699005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37567 (local address 127.0.97.2:43719)
0928 05:12:55.699959 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:55.699976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:55.700124 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:55.700439 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:55.700445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:55.700473 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:55.701309 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.701320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.702299 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.702303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.703940 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.703951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.705082 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.705089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.705268 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.705289 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:12:55.705390 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:12:55.705550 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":30,"thread_start_us":718,"threads_started":1}
I20260928 05:12:55.706091 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.698892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39433 (local address 127.0.97.2:48875)
0928 05:12:55.699850 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 05:12:55.699873 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 05:12:55.700011 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:12:55.700371 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:12:55.700377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:12:55.700403 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:12:55.700808 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.700819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.702256 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.702261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.703979 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:12:55.703990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.705117 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.705122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:12:55.705264 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.705285 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:12:55.705391 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 05:12:55.705554 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":96,"thread_start_us":500,"threads_started":1}
I20260928 05:12:55.707044 12169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e30a99b9264155b74bb758630548d6" candidate_uuid: "091c5a9e0f0640849efeb0c13dea352b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" is_pre_election: true
I20260928 05:12:55.706954 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e30a99b9264155b74bb758630548d6" candidate_uuid: "091c5a9e0f0640849efeb0c13dea352b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f06ee8bacc489d822a4d269bb47609" is_pre_election: true
W20260928 05:12:55.707191 11939 tablet.cc:2369] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:12:55.707520 12169 raft_consensus.cc:2454] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 091c5a9e0f0640849efeb0c13dea352b in term 0.
I20260928 05:12:55.707553 11892 raft_consensus.cc:2454] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 091c5a9e0f0640849efeb0c13dea352b in term 0.
I20260928 05:12:55.708169 11969 leader_election.cc:302] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091c5a9e0f0640849efeb0c13dea352b, 5f1b15b2451a4f6989b4ae0a9be405a6; no voters:
I20260928 05:12:55.708897 12241 raft_consensus.cc:2781] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:12:55.709044 12241 raft_consensus.cc:482] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:12:55.709188 12241 raft_consensus.cc:3037] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:55.711357 12241 raft_consensus.cc:504] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.711944 12241 leader_election.cc:288] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [CANDIDATE]: Term 1 election: Requested vote from peers 5f1b15b2451a4f6989b4ae0a9be405a6 (127.0.97.3:37567), d2f06ee8bacc489d822a4d269bb47609 (127.0.97.1:39433)
I20260928 05:12:55.712311 12169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e30a99b9264155b74bb758630548d6" candidate_uuid: "091c5a9e0f0640849efeb0c13dea352b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6"
I20260928 05:12:55.712390 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e30a99b9264155b74bb758630548d6" candidate_uuid: "091c5a9e0f0640849efeb0c13dea352b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f06ee8bacc489d822a4d269bb47609"
I20260928 05:12:55.712493 12169 raft_consensus.cc:3037] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:55.712604 11892 raft_consensus.cc:3037] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:12:55.714955 11892 raft_consensus.cc:2454] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091c5a9e0f0640849efeb0c13dea352b in term 1.
I20260928 05:12:55.715214 12169 raft_consensus.cc:2454] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091c5a9e0f0640849efeb0c13dea352b in term 1.
I20260928 05:12:55.715409 11971 leader_election.cc:302] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091c5a9e0f0640849efeb0c13dea352b, d2f06ee8bacc489d822a4d269bb47609; no voters:
I20260928 05:12:55.715690 12241 raft_consensus.cc:2781] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:12:55.716166 12241 raft_consensus.cc:686] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 1 LEADER]: Becoming Leader. State: Replica: 091c5a9e0f0640849efeb0c13dea352b, State: Running, Role: LEADER
I20260928 05:12:55.716586 12241 consensus_queue.cc:237] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } }
I20260928 05:12:55.720705 11724 catalog_manager.cc:5495] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b reported cstate change: term changed from 0 to 1, leader changed from <none> to 091c5a9e0f0640849efeb0c13dea352b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "091c5a9e0f0640849efeb0c13dea352b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091c5a9e0f0640849efeb0c13dea352b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 } health_report { overall_health: UNKNOWN } } }
I20260928 05:12:55.779798 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.774523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.0.97.2:40223)
0928 05:12:55.774773 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.774779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.774799 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:12:55.774974 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.774978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.775065 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.775214 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:12:55.775596 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.776301 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.777675 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.778071 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.778201 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.778714 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:12:55.779198 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:12:55.779355 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.779453 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.779528 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:12:55.787477 12169 raft_consensus.cc:1261] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 1 FOLLOWER]: Refusing update from remote peer 091c5a9e0f0640849efeb0c13dea352b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:12:55.787483 11892 raft_consensus.cc:1261] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 1 FOLLOWER]: Refusing update from remote peer 091c5a9e0f0640849efeb0c13dea352b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:12:55.788437 12244 consensus_queue.cc:1035] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1b15b2451a4f6989b4ae0a9be405a6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:12:55.788745 12241 consensus_queue.cc:1035] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f06ee8bacc489d822a4d269bb47609" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:12:55.803269 12247 mvcc.cc:205] Tried to move back new op lower bound from 7334184451216506880 to 7334184450938802176. Current Snapshot: MvccSnapshot[applied={T|T < 7334184451216506880 or (T in {7334184451216506880})}]
I20260928 05:12:55.813947 11722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:12:55.814248 11722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:12:55.817968 11722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44d5b8f2b0a74189bf07e2a07a2a593e: Sending DeleteTablet for 3 replicas of tablet 66e30a99b9264155b74bb758630548d6
I20260928 05:12:55.819078 12015 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e30a99b9264155b74bb758630548d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:12:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59462
I20260928 05:12:55.819226 12149 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e30a99b9264155b74bb758630548d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:12:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38030
I20260928 05:12:55.819202 11872 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e30a99b9264155b74bb758630548d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:12:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41320
I20260928 05:12:55.823163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 11804
I20260928 05:12:55.830864 12259 tablet_replica.cc:330] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: stopping tablet replica
I20260928 05:12:55.831393 12259 raft_consensus.cc:2229] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:12:55.832000 12259 raft_consensus.cc:2258] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:12:55.834368 12258 tablet_replica.cc:330] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: stopping tablet replica
I20260928 05:12:55.834910 12258 raft_consensus.cc:2229] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:12:55.835461 12258 raft_consensus.cc:2258] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:12:55.837940 12259 ts_tablet_manager.cc:1905] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:12:55.844837 12259 ts_tablet_manager.cc:1918] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:12:55.845016 12259 log.cc:1196] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/66e30a99b9264155b74bb758630548d6
I20260928 05:12:55.845506 12259 ts_tablet_manager.cc:1939] T 66e30a99b9264155b74bb758630548d6 P 5f1b15b2451a4f6989b4ae0a9be405a6: Deleting consensus metadata
Sep 28 05:12:55 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:12:55.848646 12258 ts_tablet_manager.cc:1905] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:12:55.849258 12257 tablet_replica.cc:330] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: stopping tablet replica
I20260928 05:12:55.849882 12257 raft_consensus.cc:2229] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:12:55.849969 11709 catalog_manager.cc:4841] TS 5f1b15b2451a4f6989b4ae0a9be405a6 (127.0.97.3:37567): tablet 66e30a99b9264155b74bb758630548d6 (table test-table [id=266ebf3cb552493789bb4dec88e2f6be]) successfully deleted
I20260928 05:12:55.850675 12257 raft_consensus.cc:2258] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:12:55.853149 12257 ts_tablet_manager.cc:1905] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:12:55.856007 12258 ts_tablet_manager.cc:1918] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:12:55.856606 12258 log.cc:1196] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/66e30a99b9264155b74bb758630548d6
I20260928 05:12:55.857121 12258 ts_tablet_manager.cc:1939] T 66e30a99b9264155b74bb758630548d6 P d2f06ee8bacc489d822a4d269bb47609: Deleting consensus metadata
I20260928 05:12:55.858930 11711 catalog_manager.cc:4841] TS d2f06ee8bacc489d822a4d269bb47609 (127.0.97.1:39433): tablet 66e30a99b9264155b74bb758630548d6 (table test-table [id=266ebf3cb552493789bb4dec88e2f6be]) successfully deleted
I20260928 05:12:55.859274 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.824058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.0.97.1:39433)
0928 05:12:55.824228 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.824234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.837481 (+ 13247us) server_negotiation.cc:407] Connection header received
0928 05:12:55.837949 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.837957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.838041 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.838114 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:55.839933 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.841028 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.843069 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.843700 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.843799 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.851579 (+ 7780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:55.851596 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:55.851630 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:55.851677 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:55.855677 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.856625 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.856635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.856639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.856781 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.857257 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.857264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.857269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.857730 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:55.858016 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.858654 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.858876 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 05:12:55.860637 11933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:55.862032 12257 ts_tablet_manager.cc:1918] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:12:55.862242 12257 log.cc:1196] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/66e30a99b9264155b74bb758630548d6
I20260928 05:12:55.862874 12257 ts_tablet_manager.cc:1939] T 66e30a99b9264155b74bb758630548d6 P 091c5a9e0f0640849efeb0c13dea352b: Deleting consensus metadata
I20260928 05:12:55.864869 11709 catalog_manager.cc:4841] TS 091c5a9e0f0640849efeb0c13dea352b (127.0.97.2:40223): tablet 66e30a99b9264155b74bb758630548d6 (table test-table [id=266ebf3cb552493789bb4dec88e2f6be]) successfully deleted
I20260928 05:12:55.951713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 11804
I20260928 05:12:55.969569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 11945
Sep 28 05:12:55 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:12:55.986899 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:55.970325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.0.97.2:40223)
0928 05:12:55.970524 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:55.970569 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:55.970677 (+ 108us) server_negotiation.cc:407] Connection header received
0928 05:12:55.970887 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:55.970890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:55.970952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:55.971028 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:55.972649 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.973495 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.975373 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:55.975770 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:55.975862 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:55.980934 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:55.980947 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:55.980963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:55.981001 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:55.984093 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.984963 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.984970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.984972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.985124 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:55.985502 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:55.985505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:55.985507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:55.985810 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:55.986035 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:55.986380 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 05:12:55.986570 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:12:55.987560 12075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:56.103345 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 11945
I20260928 05:12:56.129000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 12086
Sep 28 05:12:56 dist-test-slave-6718 krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:12:56.152096 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:12:56.130077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.0.97.3:37567)
0928 05:12:56.130245 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 05:12:56.130251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:12:56.130269 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:12:56.130336 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:12:56.130342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:12:56.130429 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:12:56.130583 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:12:56.132580 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:56.133802 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:56.138455 (+ 4653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:12:56.139158 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:12:56.139303 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:12:56.145459 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:12:56.145485 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:12:56.145509 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:12:56.145571 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:12:56.149195 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:56.149996 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:56.150005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:56.150009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:56.150143 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:12:56.150580 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:12:56.150585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:12:56.150588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:12:56.150943 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:12:56.151251 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 05:12:56.151519 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:12:56.151728 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:12:56.152765 12209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:56.241137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 12086
I20260928 05:12:56.257172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 11692
I20260928 05:12:56.258275 11754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:12:56.369323 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 11692
I20260928 05:12:56.393576 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 05:12:56.728681 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 05:12:56.728884 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 05:12:57.064996 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 05:12:57.065 UTC [11278] LOG: received smart shutdown request
2026-09-28 05:12:57.068 UTC [11278] LOG: background worker "logical replication launcher" (PID 11287) exited with exit code 1
2026-09-28 05:12:57.069 UTC [11282] LOG: shutting down
2026-09-28 05:12:57.219 UTC [11278] LOG: database system is shut down
2026-09-28T05:12:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49218 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:12:57 dist-test-slave-6718 krb5kdc[12273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:12:57 dist-test-slave-6718 krb5kdc[12273](info): set up 2 sockets
Sep 28 05:12:57 dist-test-slave-6718 krb5kdc[12273](info): commencing operation
krb5kdc: starting...
W20260928 05:12:58.462723 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:12:58 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:12:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 05:12:58.773366 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzfOJKc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/postgres -l logfile start
2026-09-28 05:13:01.139 UTC [12303] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 05:13:01.139 UTC [12303] LOG: listening on IPv4 address "127.0.97.20", port 48155
2026-09-28 05:13:01.142 UTC [12303] LOG: listening on Unix socket "/tmp/.s.PGSQL.48155"
2026-09-28 05:13:01.165 UTC [12306] LOG: database system was shut down at 2026-09-28 05:12:59 UTC
2026-09-28 05:13:01.168 UTC [12303] LOG: database system is ready to accept connections
I20260928 05:13:02.142575 388 mini_postgres.cc:96] Postgres bound to 48155
2026-09-28 05:13:02.158 UTC [12314] FATAL: database "slave" does not exist
127.0.97.20:48155 - accepting connections
I20260928 05:13:02.159534 388 mini_ranger.cc:161] Starting Ranger...
I20260928 05:13:02.188694 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 05:13:02.636693 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 05:13:02.636783 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-admin
2026-09-28 05:13:02,929 [I] DB FLAVOR :POSTGRES
2026-09-28 05:13:02,930 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:13:02,930 [I] Checking connection..
2026-09-28 05:13:02,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 05:13:03,181 [I] Checking connection passed.
2026-09-28 05:13:03,181 [I] --------- Verifying version history table ---------
2026-09-28 05:13:03,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:13:03,404 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:13:03,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:13:03,657 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 05:13:03,657 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 05:13:03,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 05:13:03.892 UTC [12378] WARNING: there is no transaction in progress
2026-09-28 05:13:03,898 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 05:13:03,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 05:13:04,137 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 05:13:04,138 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:13:04,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 05:13:04,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 05:13:04,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6718.c.gcp-upstream.internal','N') ;"
2026-09-28 05:13:04,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:13:05,143 [I] Table x_portal_user does not exist in database ranger
2026-09-28 05:13:05,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:13:05,404 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 05:13:05,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:13:05,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:13:05,874 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 05:13:05,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 05:13:06.825 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.842 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.859 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.875 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.890 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.908 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.972 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.979 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:06.994 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.005 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.015 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.030 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.040 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.046 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.053 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07.059 UTC [12486] WARNING: there is no transaction in progress
2026-09-28 05:13:07,673 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 05:13:07,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 05:13:07,916 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 05:13:07,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 05:13:08,176 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 05:13:08,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 05:13:08,419 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 05:13:08,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 05:13:08,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 05:13:08,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6718.c.gcp-upstream.internal';"
2026-09-28 05:13:09,143 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 05:13:09,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 05:13:09,396 [I] DB_PATCHES have already been applied
I20260928 05:13:09.402078 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzfOJKc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 05:13:09.402184 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 05:13:09.416477 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:36171
Sep 28, 2026 5:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:13:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 05:13:10.532226 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.373s user 0.003s sys 0.037s
Sep 28, 2026 5:13:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:13:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 5:13:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:13:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:13:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 05:13:32.500236 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 05:13:32 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572412, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 05:13:32.648236 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 05:13:32.668613 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 05:13:33.463570 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 05:13:33.463688 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms
I20260928 05:13:33.463766 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48155
2026-09-28 05:13:33,816 [I] DB FLAVOR :POSTGRES
2026-09-28 05:13:33,816 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 05:13:33,816 [I] Checking connection
2026-09-28 05:13:33,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 05:13:34,056 [I] connection success
2026-09-28 05:13:34,056 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 05:13:34,056 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 05:13:34,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 05:13:34,284 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 05:13:34,284 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 05:13:34,284 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 05:13:34,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 05:13:34,527 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 05:13:34.949733 388 mini_ranger_kms.cc:310] Created kms service
I20260928 05:13:35.128430 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 05:13:35.174224 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 05:13:35 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572412, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:35.581193 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 05:13:35.581516 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzfOJKc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzfOJKc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 05:13:35.581597 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 05:13:35.596073 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12666
I20260928 05:13:35.596203 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:55093
Sep 28, 2026 5:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 05:13:37 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572417, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:37.917541 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.269s user 0.067s sys 0.296s
I20260928 05:13:37.917797 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 05:13:37.917966 388 mini_ranger_kms.cc:401] 127.0.97.20:55093/kms/v1/keys
Sep 28 05:13:38 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572417, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:13:38 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572417, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 05:13:40 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:13:40.926827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34819
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34819
--ranger_config_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:41.253646 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:41.253921 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:41.254009 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:41.261245 12717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:41.261350 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:41.261395 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:41.261451 12717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:41.261498 12717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:41.271596 12717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:55093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34819
--ranger_config_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:41.272814 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:41.274040 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:41.286151 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:41.286109 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:41.286109 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:41.286705 12717 server_base.cc:1034] running on GCE node
I20260928 05:13:41.287485 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:41.289184 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:41.290412 12717 hybrid_clock.cc:648] HybridClock initialized: now 1790572421290362 us; error 34 us; skew 500 ppm
I20260928 05:13:41.290803 12717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:41 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:41.293823 12717 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:13:41.295258 12717 webserver.cc:460] Webserver started at http://127.0.97.62:44297/ using document root <none> and password file <none>
I20260928 05:13:41.295884 12717 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:41.296020 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:41.296321 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:13:41 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:41.353319 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "f3e812afb04f48e880a2f5d0b22bebf7"
format_stamp: "Formatted at 2026-09-28 05:13:41 on dist-test-slave-6718"
server_key: "102437683de14fa0d9ea85d0e0f5790b"
server_key_iv: "6fd94c1e34850a966778f50b76f40856"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:41.354197 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3e812afb04f48e880a2f5d0b22bebf7"
format_stamp: "Formatted at 2026-09-28 05:13:41 on dist-test-slave-6718"
server_key: "102437683de14fa0d9ea85d0e0f5790b"
server_key_iv: "6fd94c1e34850a966778f50b76f40856"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:41.359781 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:13:41.379347 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:41.381081 12717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:13:41.381270 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "f3e812afb04f48e880a2f5d0b22bebf7"
format_stamp: "Formatted at 2026-09-28 05:13:41 on dist-test-slave-6718"
server_key: "102437683de14fa0d9ea85d0e0f5790b"
server_key_iv: "6fd94c1e34850a966778f50b76f40856"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:41.381518 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:41.406674 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:41.411557 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:41.411888 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:41.431567 12717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34819
I20260928 05:13:41.431599 12783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34819 every 8 connection(s)
I20260928 05:13:41.432889 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:13:41.436471 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:41.439985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 12717
I20260928 05:13:41.440189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:13:41.440814 388 external_mini_cluster.cc:1428] Setting key 3a0e1d4217cb658af3c0affacadf5321
I20260928 05:13:41.449275 12784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Bootstrap starting.
I20260928 05:13:41.452764 12784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:41.453816 12784 log.cc:824] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:13:41 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:41.456799 12784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: No bootstrap required, opened a new log
I20260928 05:13:41.463251 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:41.442689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.97.62:34819)
0928 05:13:41.443455 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:41.443470 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:41.443518 (+ 48us) server_negotiation.cc:407] Connection header received
0928 05:13:41.445144 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:41.445173 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:41.445680 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:41.446099 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:41.447322 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:41.449219 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:41.450776 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:41.451806 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:41.451986 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:41.456339 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:41.456346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:41.456360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:41.456395 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:41.459402 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:41.460213 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:41.460218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:41.460224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:41.460376 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:41.460759 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:41.460762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:41.460763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:41.461247 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:41.461471 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:41.461898 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 05:13:41.462207 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":314,"threads_started":1}
I20260928 05:13:41.465634 12784 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:41.466271 12784 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:41.466434 12784 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e812afb04f48e880a2f5d0b22bebf7, State: Initialized, Role: FOLLOWER
I20260928 05:13:41.467100 12784 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:41.467344 12784 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:13:41.467478 12784 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:13:41.467656 12784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:41.469656 12784 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:41.470104 12784 leader_election.cc:302] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e812afb04f48e880a2f5d0b22bebf7; no voters:
I20260928 05:13:41.470702 12784 leader_election.cc:288] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:13:41.470829 12789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:13:41.471478 12789 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 LEADER]: Becoming Leader. State: Replica: f3e812afb04f48e880a2f5d0b22bebf7, State: Running, Role: LEADER
I20260928 05:13:41.472064 12789 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:41.472734 12784 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:13:41.474774 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e812afb04f48e880a2f5d0b22bebf7. Latest consensus state: current_term: 1 leader_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } } }
I20260928 05:13:41.475155 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:41.475083 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } } }
I20260928 05:13:41.475338 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:41.475951 12797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:13:41.481447 12797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:13:41.484432 12784 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 05:13:41.498639 12797 catalog_manager.cc:1347] Generated new cluster ID: 55071d8f1cfd48baad6dabd810df2db0
I20260928 05:13:41.498780 12797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:13:41.527395 12797 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:13:41.528893 12797 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:13:41.538190 12797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Generated new TSK 0
I20260928 05:13:41.539417 12797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:13:42.246973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34819
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:13:42 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 05:13:42.687420 12829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:42.687819 12829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:42.688035 12829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:42.699635 12829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:42.699875 12829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:42.700094 12829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:13:42.716020 12829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:55093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:42.717839 12829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:42.719677 12829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:42.735919 12839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:42.737591 12840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:42.744112 12842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:42.745765 12829 server_base.cc:1034] running on GCE node
I20260928 05:13:42.746585 12829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:42.747860 12829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:42.749118 12829 hybrid_clock.cc:648] HybridClock initialized: now 1790572422749039 us; error 59 us; skew 500 ppm
I20260928 05:13:42.749500 12829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:42 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:42.753419 12829 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:13:42.755108 12829 webserver.cc:460] Webserver started at http://127.0.97.1:35237/ using document root <none> and password file <none>
I20260928 05:13:42.755885 12829 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:42.756026 12829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:42.756340 12829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:13:42 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:42.789552 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ecf7a0104804308a50758f7cf0b480e"
format_stamp: "Formatted at 2026-09-28 05:13:42 on dist-test-slave-6718"
server_key: "c2b3f82e5f03e65dd9a91ef0afdb1d88"
server_key_iv: "3d77815ae9c62c59d68500b8ff7e9892"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:42.790720 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ecf7a0104804308a50758f7cf0b480e"
format_stamp: "Formatted at 2026-09-28 05:13:42 on dist-test-slave-6718"
server_key: "c2b3f82e5f03e65dd9a91ef0afdb1d88"
server_key_iv: "3d77815ae9c62c59d68500b8ff7e9892"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:42.801997 12829 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.003s
I20260928 05:13:42.835210 12849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:42.837518 12829 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.011s
I20260928 05:13:42.837800 12829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "2ecf7a0104804308a50758f7cf0b480e"
format_stamp: "Formatted at 2026-09-28 05:13:42 on dist-test-slave-6718"
server_key: "c2b3f82e5f03e65dd9a91ef0afdb1d88"
server_key_iv: "3d77815ae9c62c59d68500b8ff7e9892"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:42.838171 12829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:42.926872 12829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:42.932775 12829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:42.933169 12829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:42.938205 12829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:42.943368 12829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:42.943529 12829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:42.943674 12829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:42.943753 12829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:13:42 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:43.041915 12829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42621
I20260928 05:13:43.041931 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42621 every 8 connection(s)
I20260928 05:13:43.043356 12829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:13:43.048591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 12829
I20260928 05:13:43.048848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:13:43.049829 388 external_mini_cluster.cc:1428] Setting key e899d2047529cc77f38334da85f137a2
Sep 28 05:13:43 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:13:43.087596 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:43.051034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44131 (local address 127.0.97.62:34819)
0928 05:13:43.066815 (+ 15781us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:43.066823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:43.066845 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:13:43.066908 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:43.066915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:43.067012 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:43.067129 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:43.069269 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.070562 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.073305 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.074050 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.074243 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:43.079053 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:43.079060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:43.079065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:43.079106 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:43.083392 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:43.084622 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:43.084628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:43.084632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:43.084777 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:43.085338 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:43.085343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:43.085346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:43.085700 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:43.085818 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:43.086889 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0928 05:13:43.087124 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15634,"thread_start_us":24257,"threads_started":1}
I20260928 05:13:43.089776 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:43.047433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34819 (local address 127.0.97.1:44131)
0928 05:13:43.060149 (+ 12716us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:43.060220 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:43.063460 (+ 3240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:43.067569 (+ 4109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:43.067584 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:43.068228 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:43.069064 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:43.069092 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.070769 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.070777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:43.073140 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:43.073167 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.074609 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.074618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:43.074945 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:43.075755 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:43.075790 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:43.078781 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:43.083563 (+ 4782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:43.083571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:43.083589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:43.084478 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:43.084927 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:43.084932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:43.084936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:43.085206 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:43.085921 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:43.085929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:43.086356 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 05:13:43.087879 (+ 1523us) client_negotiation.cc:240] Negotiation successful
0928 05:13:43.088422 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12295,"thread_start_us":16362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:13:43.091768 12962 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34819
I20260928 05:13:43.092074 12962 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:43.092890 12962 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:43.095423 12749 ts_manager.cc:194] Registered new tserver with Master: 2ecf7a0104804308a50758f7cf0b480e (127.0.97.1:42621)
I20260928 05:13:43.097591 12749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:13:43.131752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34819
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:13:43 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 05:13:43.564369 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:43.564826 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:43.565019 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:43.575778 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:43.575978 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:43.576210 12968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:13:43.589228 12968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:55093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:43.590605 12968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:43.592022 12968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:43.610484 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:43.611322 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:43.619531 12968 server_base.cc:1034] running on GCE node
W20260928 05:13:43.619990 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:43.620819 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:43.628386 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:43.629887 12968 hybrid_clock.cc:648] HybridClock initialized: now 1790572423629779 us; error 109 us; skew 500 ppm
I20260928 05:13:43.631032 12968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:43 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:43.635605 12968 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:13:43.638947 12968 webserver.cc:460] Webserver started at http://127.0.97.2:35027/ using document root <none> and password file <none>
I20260928 05:13:43.640071 12968 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:43.640275 12968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:43.640712 12968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:13:43 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:43.658159 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "b30b5f0decb34297805e50b6a6987dfa"
format_stamp: "Formatted at 2026-09-28 05:13:43 on dist-test-slave-6718"
server_key: "fefb8c70c248437a112b7185dc7285b9"
server_key_iv: "2c25bc2b90a58e4f2028e39639b1f291"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:43.659231 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b30b5f0decb34297805e50b6a6987dfa"
format_stamp: "Formatted at 2026-09-28 05:13:43 on dist-test-slave-6718"
server_key: "fefb8c70c248437a112b7185dc7285b9"
server_key_iv: "2c25bc2b90a58e4f2028e39639b1f291"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:43.665968 12968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 05:13:43.684592 12992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:43.686537 12968 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260928 05:13:43.686746 12968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "b30b5f0decb34297805e50b6a6987dfa"
format_stamp: "Formatted at 2026-09-28 05:13:43 on dist-test-slave-6718"
server_key: "fefb8c70c248437a112b7185dc7285b9"
server_key_iv: "2c25bc2b90a58e4f2028e39639b1f291"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:43.687007 12968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:43.705574 12968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:43.710601 12968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:43.710938 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:43.712087 12968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:43.713912 12968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:43.714011 12968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:43.714138 12968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:43.714217 12968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:43.748844 12968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40065
I20260928 05:13:43.748863 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40065 every 8 connection(s)
I20260928 05:13:43.750461 12968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:13:43.751531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 12968
I20260928 05:13:43.751755 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:13:43.752305 388 external_mini_cluster.cc:1428] Setting key d4d1a65ae86269503b015baff658af93
Sep 28 05:13:43 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 05:13:43.791530 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:43.754801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57619 (local address 127.0.97.62:34819)
0928 05:13:43.771068 (+ 16267us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:43.771079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:43.774189 (+ 3110us) server_negotiation.cc:407] Connection header received
0928 05:13:43.774253 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:43.774257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:43.774339 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:43.774470 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:43.776212 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.777069 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.778929 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.779377 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.779546 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:43.783692 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:43.783703 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:43.783711 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:43.783764 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:43.787711 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:43.788756 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:43.788761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:43.788764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:43.788904 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:43.789394 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:43.789401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:43.789404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:43.789887 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:43.790055 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:43.790871 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 05:13:43.791112 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16058,"thread_start_us":15949,"threads_started":1}
I20260928 05:13:43.792860 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:43.754028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34819 (local address 127.0.97.2:57619)
0928 05:13:43.771727 (+ 17699us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:43.771792 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:43.773850 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:43.774799 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:43.774815 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:43.775365 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:43.776009 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:43.776029 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.777238 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.777246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:43.778767 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:43.778780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:43.779754 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:43.779760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:43.779980 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:43.780736 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:43.780761 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:43.783390 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:43.787866 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:43.787876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:43.787894 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:43.788603 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:43.789044 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:43.789050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:43.789054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:43.789263 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:43.790029 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:43.790064 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:43.790418 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 05:13:43.790764 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 05:13:43.791345 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17384,"thread_start_us":17132,"threads_started":1}
I20260928 05:13:43.794553 13105 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34819
I20260928 05:13:43.794898 13105 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:43.795670 13105 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:43.797264 12749 ts_manager.cc:194] Registered new tserver with Master: b30b5f0decb34297805e50b6a6987dfa (127.0.97.2:40065)
I20260928 05:13:43.798728 12749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:13:43.819999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34819
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:13:44.101480 12962 heartbeater.cc:503] Master 127.0.97.62:34819 was elected leader, sending a full tablet report...
W20260928 05:13:44.153244 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:44.153499 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:44.153586 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:44.160969 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:44.161082 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:44.161191 13111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:13:44.171043 13111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:55093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:44.172268 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:44.173727 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:44.186934 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:44.186993 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:44.186998 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:44.187784 13111 server_base.cc:1034] running on GCE node
I20260928 05:13:44.188310 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:44.189203 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:44.190395 13111 hybrid_clock.cc:648] HybridClock initialized: now 1790572424190361 us; error 32 us; skew 500 ppm
I20260928 05:13:44.190737 13111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:44 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:44.193784 13111 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:13:44.195158 13111 webserver.cc:460] Webserver started at http://127.0.97.3:36709/ using document root <none> and password file <none>
I20260928 05:13:44.195799 13111 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:44.195924 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:44.196220 13111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 05:13:44 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:44.209724 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c"
format_stamp: "Formatted at 2026-09-28 05:13:44 on dist-test-slave-6718"
server_key: "1468448b435f79d6908882fd7a16c7d7"
server_key_iv: "6159d82dfbd3260448debf4c5e491ae7"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:44.210501 13111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c"
format_stamp: "Formatted at 2026-09-28 05:13:44 on dist-test-slave-6718"
server_key: "1468448b435f79d6908882fd7a16c7d7"
server_key_iv: "6159d82dfbd3260448debf4c5e491ae7"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:44.215689 13111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:13:44.229518 13126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:44.231066 13111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:13:44.231247 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c"
format_stamp: "Formatted at 2026-09-28 05:13:44 on dist-test-slave-6718"
server_key: "1468448b435f79d6908882fd7a16c7d7"
server_key_iv: "6159d82dfbd3260448debf4c5e491ae7"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:44.231498 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:44.245311 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:44.248955 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:44.249255 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:44.250176 13111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:44.251540 13111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:44.251631 13111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:44.251757 13111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:44.251835 13111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:44.285036 13111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44405
I20260928 05:13:44.285085 13238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44405 every 8 connection(s)
I20260928 05:13:44.286304 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:13:44.293330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13111
I20260928 05:13:44.293510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:13:44.294028 388 external_mini_cluster.cc:1428] Setting key 3e426ea1697553fcbaa2a8d7503cedfd
Sep 28 05:13:44 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:44.305857 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:44.288920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54401 (local address 127.0.97.62:34819)
0928 05:13:44.289116 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:44.289122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:44.290270 (+ 1148us) server_negotiation.cc:407] Connection header received
0928 05:13:44.291961 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:44.291966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:44.292060 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:44.292149 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:44.293701 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.294734 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.296754 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.297310 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.297413 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:44.300698 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:44.300703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:44.300706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:44.300738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:44.303306 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.304003 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.304006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.304008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.304119 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.304494 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.304497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.304499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.304743 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:44.304848 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:44.305467 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 05:13:44.305633 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:13:44.306948 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:44.289428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34819 (local address 127.0.97.3:54401)
0928 05:13:44.290121 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:44.290161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:44.291718 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:44.292332 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:44.292339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:44.292726 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:44.293440 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:44.293467 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.294899 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.294904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:44.296604 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:44.296622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.297493 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.297498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:44.297718 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:44.298313 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:44.298334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:44.300487 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:44.303450 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:44.303457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:44.303470 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:44.303903 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:44.304234 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:44.304238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:44.304241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:44.304404 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:44.304845 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:44.304851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:44.305160 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 05:13:44.305388 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 05:13:44.305764 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":295,"threads_started":1}
I20260928 05:13:44.308419 13239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34819
I20260928 05:13:44.308734 13239 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:44.309422 13239 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:44.310793 12749 ts_manager.cc:194] Registered new tserver with Master: 8aaee7c8de5e4d568c1796d3ceeb3e3c (127.0.97.3:44405)
I20260928 05:13:44.311575 12749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54401
I20260928 05:13:44.321677 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:13:44.323375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 12829
Sep 28 05:13:44 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:13:44.339756 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:44.324259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.0.97.1:42621)
0928 05:13:44.324865 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:44.324872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:44.324897 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:13:44.324951 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:44.324957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:44.325162 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:44.325291 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:44.326891 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.327882 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.329230 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.329739 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.329830 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:44.334327 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:44.334338 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:44.334352 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:44.334383 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:44.337349 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.337989 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.337995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.337997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.338129 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.338445 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.338447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.338449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.338798 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:44.339013 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:44.339287 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 05:13:44.339471 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
I20260928 05:13:44.340965 12957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:44.422384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 12829
I20260928 05:13:44.438097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 12968
Sep 28 05:13:44 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:13:44.453521 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:44.438984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.97.2:40065)
0928 05:13:44.439480 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:44.439487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:44.439511 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:13:44.439584 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:44.439589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:44.439840 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:44.439995 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:44.441215 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.442164 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.443522 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.444049 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.444144 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:44.448286 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:44.448297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:44.448311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:44.448341 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:44.450969 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.451650 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.451657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.451659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.451795 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.452238 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.452242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.452244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.452553 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:44.452781 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:44.453044 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 05:13:44.453250 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":253,"threads_started":1}
I20260928 05:13:44.454769 13100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:44.539139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 12968
I20260928 05:13:44.555002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13111
Sep 28 05:13:44 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:13:44.571043 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:44.555889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.0.97.3:44405)
0928 05:13:44.556478 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:44.556485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:44.556512 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:13:44.556567 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:44.556573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:44.556877 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:44.557000 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:44.558278 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.559361 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.560667 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:44.561199 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:44.561295 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:44.565527 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:44.565543 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:44.565557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:44.565597 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:44.568385 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.568997 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.569003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.569006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.569130 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:44.569479 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:44.569482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:44.569484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:44.569795 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:44.570015 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:44.570285 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 05:13:44.570462 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":315,"threads_started":1}
I20260928 05:13:44.572459 13234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:44.654060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13111
I20260928 05:13:44.669399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 12717
I20260928 05:13:44.670710 12779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:44.776461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 12717
I20260928 05:13:44.799919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34819
--webserver_interface=127.0.97.62
--webserver_port=44297
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34819
--ranger_config_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:45.116369 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:45.116685 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:45.116851 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:45.123901 13250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:45.124037 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:45.124154 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:45.124248 13250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:45.124336 13250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:45.133884 13250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34819
--ranger_config_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:45.135186 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:45.136508 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:45.148238 13256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:45.148232 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:45.148272 13258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:45.149029 13250 server_base.cc:1034] running on GCE node
I20260928 05:13:45.149812 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:45.151304 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:45.152505 13250 hybrid_clock.cc:648] HybridClock initialized: now 1790572425152464 us; error 37 us; skew 500 ppm
I20260928 05:13:45.152855 13250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:45 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:45.155916 13250 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:13:45.157339 13250 webserver.cc:460] Webserver started at http://127.0.97.62:44297/ using document root <none> and password file <none>
I20260928 05:13:45.157991 13250 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:45.158174 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:13:45 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:45.213135 13250 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:13:45.216380 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:45.217759 13250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:13:45.217955 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "f3e812afb04f48e880a2f5d0b22bebf7"
format_stamp: "Formatted at 2026-09-28 05:13:41 on dist-test-slave-6718"
server_key: "102437683de14fa0d9ea85d0e0f5790b"
server_key_iv: "6fd94c1e34850a966778f50b76f40856"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:45.219017 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:45.231717 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:45.235198 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:45.235476 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:45.253753 13250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34819
I20260928 05:13:45.253818 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34819 every 8 connection(s)
I20260928 05:13:45.255368 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:13:45.262006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13250
I20260928 05:13:45.263216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42621
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34819
--webserver_port=35237
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:13:45.263339 13318 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:13:45.269088 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Bootstrap starting.
I20260928 05:13:45.290120 13318 log.cc:824] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:45.298818 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:13:45.299358 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Bootstrap complete.
I20260928 05:13:45.308100 13318 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:45.308910 13318 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e812afb04f48e880a2f5d0b22bebf7, State: Initialized, Role: FOLLOWER
I20260928 05:13:45.309505 13318 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:45.309732 13318 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:13:45.309847 13318 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:13:45.310029 13318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:13:45.313721 13318 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:45.314195 13318 leader_election.cc:302] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e812afb04f48e880a2f5d0b22bebf7; no voters:
I20260928 05:13:45.314848 13318 leader_election.cc:288] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:13:45.314996 13322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:13:45.316082 13322 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [term 2 LEADER]: Becoming Leader. State: Replica: f3e812afb04f48e880a2f5d0b22bebf7, State: Running, Role: LEADER
I20260928 05:13:45.316694 13322 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } }
I20260928 05:13:45.317009 13318 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:13:45.319123 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e812afb04f48e880a2f5d0b22bebf7. Latest consensus state: current_term: 2 leader_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } } }
I20260928 05:13:45.319518 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:45.320744 13323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e812afb04f48e880a2f5d0b22bebf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34819 } } }
I20260928 05:13:45.321018 13323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:45.327934 13329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:13:45.333573 13329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:13:45.334467 13318 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 05:13:45.335027 13329 catalog_manager.cc:1259] Loaded cluster ID: 55071d8f1cfd48baad6dabd810df2db0
I20260928 05:13:45.335178 13329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:13:45.339097 13329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:13:45.340516 13329 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Loaded TSK: 0
I20260928 05:13:45.342013 13329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:13:45.670359 13320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:45.670935 13320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:45.671182 13320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:45.684386 13320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:45.684633 13320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:45.684908 13320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:13:45.701467 13320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35237
--tserver_master_addrs=127.0.97.62:34819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:45.703464 13320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:45.705214 13320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:45.730952 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:45.737155 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:45.750794 13320 server_base.cc:1034] running on GCE node
W20260928 05:13:45.751909 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:45.752952 13320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:45.761330 13320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:45.765240 13320 hybrid_clock.cc:648] HybridClock initialized: now 1790572425765139 us; error 103 us; skew 500 ppm
I20260928 05:13:45.765774 13320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:45 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:45.770463 13320 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:13:45.772701 13320 webserver.cc:460] Webserver started at http://127.0.97.1:35237/ using document root <none> and password file <none>
I20260928 05:13:45.773746 13320 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:45.773964 13320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:13:45 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:45.816602 13320 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.007s sys 0.004s
I20260928 05:13:45.820277 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:45.822083 13320 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:13:45.822260 13320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "2ecf7a0104804308a50758f7cf0b480e"
format_stamp: "Formatted at 2026-09-28 05:13:42 on dist-test-slave-6718"
server_key: "c2b3f82e5f03e65dd9a91ef0afdb1d88"
server_key_iv: "3d77815ae9c62c59d68500b8ff7e9892"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:45.823689 13320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:45.879170 13320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:45.887760 13320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:45.888055 13320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:45.895394 13320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:45.897782 13320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:45.897912 13320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:45.898030 13320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:45.898150 13320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:45.955915 13320 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42621
I20260928 05:13:45.956085 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42621 every 8 connection(s)
I20260928 05:13:45.957777 13320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:13:45.965667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13320
I20260928 05:13:45.966449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40065
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34819
--webserver_port=35027
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:13:45 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:46.006294 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:45.961307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49581 (local address 127.0.97.62:34819)
0928 05:13:45.975393 (+ 14086us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:45.975422 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:45.977786 (+ 2364us) server_negotiation.cc:407] Connection header received
0928 05:13:45.982441 (+ 4655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:45.982467 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:45.983049 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:45.983492 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:45.986019 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:45.988367 (+ 2348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:45.991308 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:45.992520 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:45.993683 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:45.997455 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:45.997466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:45.997485 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:45.997523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:46.001140 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:46.002038 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:46.002046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:46.002055 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:46.002251 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:46.002776 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:46.002781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:46.002784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:46.003385 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:46.003588 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:46.004546 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 05:13:46.004945 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13764,"thread_start_us":13481,"threads_started":1}
I20260928 05:13:46.006322 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:45.961405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34819 (local address 127.0.97.1:49581)
0928 05:13:45.977524 (+ 16119us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:45.977599 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:45.981542 (+ 3943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:45.984348 (+ 2806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:45.984359 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:45.984987 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:45.985795 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:45.985821 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:45.988596 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:45.988604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:45.991052 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:45.991082 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:45.992803 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:45.992814 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:45.993163 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:45.994005 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:45.994040 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:45.997054 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:46.001297 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:46.001306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:46.001322 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:46.001883 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:46.002373 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:46.002378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:46.002382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:46.002651 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:46.003557 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:46.003566 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:46.003981 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 05:13:46.004295 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 05:13:46.005089 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15724,"thread_start_us":16327,"threads_started":1}
I20260928 05:13:46.009990 13480 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34819
I20260928 05:13:46.010321 13480 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:46.011232 13480 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
W20260928 05:13:46.014249 13480 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34819 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:13:46.014456 13480 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:46.014741 13480 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:46.015965 13480 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:46.016290 13480 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
W20260928 05:13:46.017350 13480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:13:46.400267 13484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:46.400570 13484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:46.400743 13484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:46.408195 13484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:46.408332 13484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:46.408504 13484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:13:46.418669 13484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35027
--tserver_master_addrs=127.0.97.62:34819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:46.420045 13484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:46.421656 13484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:46.439054 13498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:46.442894 13499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:46.443826 13501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:46.444725 13484 server_base.cc:1034] running on GCE node
I20260928 05:13:46.445349 13484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:46.446363 13484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:46.447579 13484 hybrid_clock.cc:648] HybridClock initialized: now 1790572426447518 us; error 55 us; skew 500 ppm
I20260928 05:13:46.447906 13484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:46 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:46.455312 13484 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:13:46.462716 13484 webserver.cc:460] Webserver started at http://127.0.97.2:35027/ using document root <none> and password file <none>
I20260928 05:13:46.464036 13484 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:46.464212 13484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:13:46 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:46.500329 13484 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.000s
I20260928 05:13:46.515319 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:46.517820 13484 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.009s
I20260928 05:13:46.518178 13484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "b30b5f0decb34297805e50b6a6987dfa"
format_stamp: "Formatted at 2026-09-28 05:13:43 on dist-test-slave-6718"
server_key: "fefb8c70c248437a112b7185dc7285b9"
server_key_iv: "2c25bc2b90a58e4f2028e39639b1f291"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:46.520504 13484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:46.590577 13484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:46.595048 13484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:46.595371 13484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:46.596563 13484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:46.598194 13484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:46.598308 13484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:46.598435 13484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:46.598554 13484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 05:13:46 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790572426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:46.665509 13484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40065
I20260928 05:13:46.665563 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40065 every 8 connection(s)
I20260928 05:13:46.666977 13484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:13:46.676702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13484
I20260928 05:13:46.677466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44405
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34819
--webserver_port=36709
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:13:46 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:46.692911 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:46.670491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34819 (local address 127.0.97.2:55153)
0928 05:13:46.671567 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:46.671634 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:46.674163 (+ 2529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:46.675034 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:46.675046 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:46.675647 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:46.676505 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:46.676527 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:46.677766 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:46.677771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:46.679189 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:46.679200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:46.679999 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:46.680006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:46.680234 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:46.680913 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:46.680949 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:46.683497 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:46.688220 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:46.688231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:46.688250 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:46.688845 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:46.689284 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:46.689291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:46.689298 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:46.689513 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:46.690184 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:46.690195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:46.690635 (+ 440us) client_negotiation.cc:769] Sending connection context
0928 05:13:46.690939 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 05:13:46.691491 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":637,"threads_started":1}
I20260928 05:13:46.694569 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:46.671162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55153 (local address 127.0.97.62:34819)
0928 05:13:46.673103 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:46.673111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:46.673137 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:13:46.674514 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:46.674521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:46.674641 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:46.674767 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:46.676694 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:46.677531 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:46.679341 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:46.679756 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:46.679901 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:46.683741 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:46.683750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:46.683758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:46.683803 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:46.688074 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:46.688997 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:46.689006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:46.689012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:46.689163 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:46.689653 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:46.689660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:46.689667 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:46.690032 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:46.693692 (+ 3660us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:46.693835 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 05:13:46.694084 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1808,"thread_start_us":1707,"threads_started":1}
I20260928 05:13:46.695746 13624 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34819
I20260928 05:13:46.696082 13624 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:46.696904 13624 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:46.698969 13283 ts_manager.cc:194] Registered new tserver with Master: b30b5f0decb34297805e50b6a6987dfa (127.0.97.2:40065)
I20260928 05:13:46.701450 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55153
I20260928 05:13:47.017519 13480 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:47.018006 13480 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:47.019866 13283 ts_manager.cc:194] Registered new tserver with Master: 2ecf7a0104804308a50758f7cf0b480e (127.0.97.1:42621)
I20260928 05:13:47.021088 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49581
Sep 28 05:13:47 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 05:13:47.093466 13629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:47.093847 13629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:47.093990 13629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:47.105239 13629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:47.105384 13629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:47.105559 13629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:13:47.120666 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:55093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36709
--tserver_master_addrs=127.0.97.62:34819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:47.122274 13629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:47.123903 13629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:47.142251 13634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:47.148072 13635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:47.156490 13629 server_base.cc:1034] running on GCE node
W20260928 05:13:47.156777 13637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:47.157842 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:47.159288 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:47.160615 13629 hybrid_clock.cc:648] HybridClock initialized: now 1790572427160559 us; error 36 us; skew 500 ppm
I20260928 05:13:47.161042 13629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:47 dist-test-slave-6718 krb5kdc[12273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:47.165256 13629 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:13:47.167166 13629 webserver.cc:460] Webserver started at http://127.0.97.3:36709/ using document root <none> and password file <none>
I20260928 05:13:47.167959 13629 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:47.168119 13629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 05:13:47 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:47.192162 13629 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260928 05:13:47.195428 13645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:47.196955 13629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:13:47.197144 13629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c"
format_stamp: "Formatted at 2026-09-28 05:13:44 on dist-test-slave-6718"
server_key: "1468448b435f79d6908882fd7a16c7d7"
server_key_iv: "6159d82dfbd3260448debf4c5e491ae7"
server_key_version: "kuduclusterkey@0"
I20260928 05:13:47.198192 13629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:47.216255 13629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:47.220461 13629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:47.220748 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:47.221760 13629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:47.223356 13629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:47.223470 13629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:47.223608 13629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:47.223693 13629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:47.282325 13629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44405
I20260928 05:13:47.282408 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44405 every 8 connection(s)
I20260928 05:13:47.284092 13629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:13:47.289746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13629
Sep 28 05:13:47 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:47.328955 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.287115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54253 (local address 127.0.97.62:34819)
0928 05:13:47.308890 (+ 21775us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.308899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.308930 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:13:47.309013 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.309021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.309124 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.309332 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:47.312221 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.313545 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.316390 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.317057 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.317172 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.321677 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:47.321684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:47.321688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:47.321723 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:47.325221 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.326178 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.326183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.326187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.326352 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.326887 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.326895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.326900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.327267 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:47.327468 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.328285 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.328538 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21604,"thread_start_us":28061,"threads_started":1}
I20260928 05:13:47.330144 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.287872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34819 (local address 127.0.97.3:54253)
0928 05:13:47.292534 (+ 4662us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.292598 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:47.294888 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:47.309692 (+ 14804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:47.309707 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:47.310341 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:47.311979 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.312007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.313752 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.313757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.316219 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.316234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.317475 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.317484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.317777 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.318576 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:47.318607 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:47.321415 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:47.325362 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:47.325371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:47.325389 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:47.326036 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:47.326482 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:47.326487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:47.326490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:47.326751 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:47.327454 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:47.327462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:47.327870 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 05:13:47.328158 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 05:13:47.328823 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4376,"thread_start_us":4155,"threads_started":1}
I20260928 05:13:47.331893 13758 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34819
I20260928 05:13:47.332208 13758 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:47.332424 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.290816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.97.62:34819)
0928 05:13:47.315426 (+ 24610us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.315430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.315445 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:13:47.315482 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.315486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.315535 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.315662 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:47.317747 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.318614 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.320682 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.321107 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.321244 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.325362 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:47.325369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:47.325374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:47.325410 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:47.328563 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.330040 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.330047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.330050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.330203 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.330812 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.330818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.330820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.331159 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:47.331420 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.331725 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.331931 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24499,"thread_start_us":17734,"threads_started":1}
I20260928 05:13:47.333031 13758 heartbeater.cc:511] Master 127.0.97.62:34819 requested a full tablet report, sending...
I20260928 05:13:47.335070 13282 ts_manager.cc:194] Registered new tserver with Master: 8aaee7c8de5e4d568c1796d3ceeb3e3c (127.0.97.3:44405)
I20260928 05:13:47.336273 13282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54253
I20260928 05:13:47.347965 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:13:47.367947 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.351101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.97.62:34819)
0928 05:13:47.351287 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.351291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.351949 (+ 658us) server_negotiation.cc:407] Connection header received
0928 05:13:47.352229 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.352234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.352315 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.352390 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:47.354430 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.355421 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.357288 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.357881 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.357991 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.361309 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:47.361316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:47.361320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:47.361359 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:47.364414 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.365204 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.365210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.365213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.365376 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.365897 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.365901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.365904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.366218 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:47.366448 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.367391 (+ 943us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.367612 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 05:13:47 dist-test-slave-6718 krb5kdc[12273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 05:13:47.374102 13282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:13:47.383519 13282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:13:47.472452 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.408895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.97.2:40065)
0928 05:13:47.433573 (+ 24678us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.433588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.434721 (+ 1133us) server_negotiation.cc:407] Connection header received
0928 05:13:47.435380 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.435393 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.435747 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.435981 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:47.437204 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.439311 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.463108 (+ 23797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.465247 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.465412 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.465702 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.470681 (+ 4979us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.472092 (+ 1411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24448,"thread_start_us":24119,"threads_started":1}
I20260928 05:13:47.477360 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.408193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40065 (local address 127.0.0.1:44248)
0928 05:13:47.434352 (+ 26159us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.434449 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:47.434958 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:47.436340 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:47.436351 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:47.436397 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:47.436930 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.436947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.439529 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.439556 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.462691 (+ 23135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.462708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.469485 (+ 6777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.469494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.469819 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.469905 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 05:13:47.470082 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 05:13:47.470371 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25767,"thread_start_us":34353,"threads_started":1}
I20260928 05:13:47.482780 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8a427bac134155be47f9b5bba702f2 (DEFAULT_TABLE table=test-table [id=1da93d208a24477abd16a5d19f9cf65a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:13:47.484287 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8a427bac134155be47f9b5bba702f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:47.494292 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.450554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41418)
0928 05:13:47.456935 (+ 6381us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.456965 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:47.457168 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:47.461450 (+ 4282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:47.461458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:47.461496 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:47.461934 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.461950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.473437 (+ 11487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.473445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.476951 (+ 3506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.476967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.493467 (+ 16500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.493473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.493651 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.493687 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:13:47.493785 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 05:13:47.493930 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6188,"thread_start_us":6718,"threads_started":1}
I20260928 05:13:47.494342 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.444371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.0.97.1:42621)
0928 05:13:47.459857 (+ 15486us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.459871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.459911 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:13:47.460046 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.460058 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.460424 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.460662 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:47.466647 (+ 5985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.473286 (+ 6639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.487293 (+ 14007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.488870 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.489006 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.489223 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.493848 (+ 4625us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.494015 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15242,"thread_start_us":16525,"threads_started":1}
I20260928 05:13:47.496860 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.457739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44405 (local address 127.0.0.1:54308)
0928 05:13:47.458997 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.459027 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:47.459190 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:47.471887 (+ 12697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:47.471893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:47.471923 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:47.477137 (+ 5214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.477148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.491178 (+ 14030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.491183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.493184 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.493199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.496106 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.496112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.496267 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.496300 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:13:47.496386 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 05:13:47.496516 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":2821,"threads_started":1}
I20260928 05:13:47.497588 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.458188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.0.97.3:44405)
0928 05:13:47.471180 (+ 12992us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.471189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.471226 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:13:47.471330 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.471340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.471618 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.471854 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:47.489345 (+ 17491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.490998 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.493368 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.495576 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.495742 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.495967 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.497108 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.497269 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12799,"thread_start_us":13878,"threads_started":1}
I20260928 05:13:47.498272 13414 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8a427bac134155be47f9b5bba702f2 (DEFAULT_TABLE table=test-table [id=1da93d208a24477abd16a5d19f9cf65a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:13:47.499806 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8a427bac134155be47f9b5bba702f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:47.504830 13693 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8a427bac134155be47f9b5bba702f2 (DEFAULT_TABLE table=test-table [id=1da93d208a24477abd16a5d19f9cf65a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:13:47.505908 13693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8a427bac134155be47f9b5bba702f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:47.521500 13781 tablet_bootstrap.cc:492] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Bootstrap starting.
I20260928 05:13:47.524739 13782 tablet_bootstrap.cc:492] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Bootstrap starting.
I20260928 05:13:47.532226 13781 tablet_bootstrap.cc:654] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:47.533568 13781 log.cc:824] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:47.534953 13782 tablet_bootstrap.cc:654] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:47.536326 13782 log.cc:824] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:47.544384 13783 tablet_bootstrap.cc:492] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: Bootstrap starting.
I20260928 05:13:47.549528 13781 tablet_bootstrap.cc:492] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: No bootstrap required, opened a new log
I20260928 05:13:47.549850 13781 ts_tablet_manager.cc:1397] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.000s
I20260928 05:13:47.551617 13783 tablet_bootstrap.cc:654] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:47.556301 13782 tablet_bootstrap.cc:492] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: No bootstrap required, opened a new log
I20260928 05:13:47.556602 13782 ts_tablet_manager.cc:1397] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20260928 05:13:47.557811 13783 log.cc:824] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:47.570871 13783 tablet_bootstrap.cc:492] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: No bootstrap required, opened a new log
I20260928 05:13:47.571266 13783 ts_tablet_manager.cc:1397] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.006s
I20260928 05:13:47.575855 13781 raft_consensus.cc:348] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.576333 13781 raft_consensus.cc:374] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:47.576442 13781 raft_consensus.cc:729] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30b5f0decb34297805e50b6a6987dfa, State: Initialized, Role: FOLLOWER
I20260928 05:13:47.577397 13782 raft_consensus.cc:348] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.577852 13782 raft_consensus.cc:374] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:47.577951 13782 raft_consensus.cc:729] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aaee7c8de5e4d568c1796d3ceeb3e3c, State: Initialized, Role: FOLLOWER
I20260928 05:13:47.578632 13782 consensus_queue.cc:260] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.579463 13781 consensus_queue.cc:260] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.592572 13781 ts_tablet_manager.cc:1428] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Time spent starting tablet: real 0.043s user 0.015s sys 0.004s
I20260928 05:13:47.593011 13624 heartbeater.cc:503] Master 127.0.97.62:34819 was elected leader, sending a full tablet report...
I20260928 05:13:47.593436 13758 heartbeater.cc:503] Master 127.0.97.62:34819 was elected leader, sending a full tablet report...
I20260928 05:13:47.595649 13782 ts_tablet_manager.cc:1428] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Time spent starting tablet: real 0.039s user 0.013s sys 0.007s
I20260928 05:13:47.594257 13783 raft_consensus.cc:348] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.597941 13783 raft_consensus.cc:374] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:47.606678 13783 raft_consensus.cc:729] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecf7a0104804308a50758f7cf0b480e, State: Initialized, Role: FOLLOWER
I20260928 05:13:47.607677 13783 consensus_queue.cc:260] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.627398 13783 ts_tablet_manager.cc:1428] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: Time spent starting tablet: real 0.056s user 0.028s sys 0.004s
I20260928 05:13:47.627516 13480 heartbeater.cc:503] Master 127.0.97.62:34819 was elected leader, sending a full tablet report...
I20260928 05:13:47.657083 13789 raft_consensus.cc:482] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:13:47.657418 13789 raft_consensus.cc:504] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.659703 13789 leader_election.cc:288] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b30b5f0decb34297805e50b6a6987dfa (127.0.97.2:40065), 8aaee7c8de5e4d568c1796d3ceeb3e3c (127.0.97.3:44405)
W20260928 05:13:47.674602 13625 tablet.cc:2369] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:47.702227 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.660672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58109 (local address 127.0.97.2:40065)
0928 05:13:47.660874 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.660881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.693946 (+ 33065us) server_negotiation.cc:407] Connection header received
0928 05:13:47.694492 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.694498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.694599 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.694668 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:47.695959 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.697057 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.699770 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.700993 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.701154 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.701258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.701821 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.701960 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:13:47.703009 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.660192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40065 (local address 127.0.97.1:58109)
0928 05:13:47.693824 (+ 33632us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.693850 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:47.694270 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:47.694877 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:47.694882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:47.694909 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:47.695233 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.695245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.697263 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.697268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.699528 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.699568 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.701418 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.701425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.701624 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.701659 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:13:47.702492 (+ 833us) client_negotiation.cc:240] Negotiation successful
0928 05:13:47.702681 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33500,"thread_start_us":32504,"threads_started":1}
I20260928 05:13:47.707048 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8a427bac134155be47f9b5bba702f2" candidate_uuid: "2ecf7a0104804308a50758f7cf0b480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b30b5f0decb34297805e50b6a6987dfa" is_pre_election: true
I20260928 05:13:47.707685 13579 raft_consensus.cc:2454] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf7a0104804308a50758f7cf0b480e in term 0.
I20260928 05:13:47.708901 13369 leader_election.cc:302] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf7a0104804308a50758f7cf0b480e, b30b5f0decb34297805e50b6a6987dfa; no voters:
I20260928 05:13:47.709407 13789 raft_consensus.cc:2781] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:13:47.709556 13789 raft_consensus.cc:482] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:13:47.709661 13789 raft_consensus.cc:3037] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:47.709745 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.693439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47259 (local address 127.0.97.3:44405)
0928 05:13:47.693683 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.693690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.703342 (+ 9652us) server_negotiation.cc:407] Connection header received
0928 05:13:47.703582 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.703588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.703661 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.703718 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:47.704248 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.705248 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.707460 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.708599 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.708694 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.708789 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.709366 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.709502 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:13:47.710439 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.693660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44405 (local address 127.0.97.1:47259)
0928 05:13:47.703248 (+ 9588us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.703267 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:47.703442 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:47.703904 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:47.703909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:47.703933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:47.704101 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.704110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.705419 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.705424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.707284 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:47.707298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.708965 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.708971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:47.709175 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.709209 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 05:13:47.709918 (+ 709us) client_negotiation.cc:240] Negotiation successful
0928 05:13:47.710087 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9487,"mutex_wait_us":307,"thread_start_us":11823,"threads_started":1}
I20260928 05:13:47.713011 13789 raft_consensus.cc:504] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.713791 13789 leader_election.cc:288] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [CANDIDATE]: Term 1 election: Requested vote from peers b30b5f0decb34297805e50b6a6987dfa (127.0.97.2:40065), 8aaee7c8de5e4d568c1796d3ceeb3e3c (127.0.97.3:44405)
I20260928 05:13:47.714725 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8a427bac134155be47f9b5bba702f2" candidate_uuid: "2ecf7a0104804308a50758f7cf0b480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b30b5f0decb34297805e50b6a6987dfa"
I20260928 05:13:47.714967 13579 raft_consensus.cc:3037] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:47.718247 13579 raft_consensus.cc:2454] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf7a0104804308a50758f7cf0b480e in term 1.
I20260928 05:13:47.719506 13369 leader_election.cc:302] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf7a0104804308a50758f7cf0b480e, b30b5f0decb34297805e50b6a6987dfa; no voters:
I20260928 05:13:47.719833 13789 raft_consensus.cc:2781] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:13:47.719660 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8a427bac134155be47f9b5bba702f2" candidate_uuid: "2ecf7a0104804308a50758f7cf0b480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" is_pre_election: true
I20260928 05:13:47.720278 13713 raft_consensus.cc:2454] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf7a0104804308a50758f7cf0b480e in term 0.
I20260928 05:13:47.720925 13712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8a427bac134155be47f9b5bba702f2" candidate_uuid: "2ecf7a0104804308a50758f7cf0b480e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c"
I20260928 05:13:47.721207 13712 raft_consensus.cc:3037] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:47.725284 13712 raft_consensus.cc:2454] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf7a0104804308a50758f7cf0b480e in term 1.
I20260928 05:13:47.726154 13789 raft_consensus.cc:686] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 1 LEADER]: Becoming Leader. State: Replica: 2ecf7a0104804308a50758f7cf0b480e, State: Running, Role: LEADER
I20260928 05:13:47.726751 13789 consensus_queue.cc:237] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } }
I20260928 05:13:47.732312 13282 catalog_manager.cc:5495] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecf7a0104804308a50758f7cf0b480e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2ecf7a0104804308a50758f7cf0b480e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ecf7a0104804308a50758f7cf0b480e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 } health_report { overall_health: UNKNOWN } } }
W20260928 05:13:47.734835 13481 tablet.cc:2369] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:47.785907 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.780966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.0.97.1:42621)
0928 05:13:47.781142 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.781146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.781164 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:13:47.781308 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.781312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.781408 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.781530 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:13:47.781888 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.782663 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.783982 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.784380 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.784467 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.784886 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:13:47.785404 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:13:47.785514 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.785596 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.785665 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 05:13:47.792510 13759 tablet.cc:2369] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:47.793828 13579 raft_consensus.cc:1261] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 1 FOLLOWER]: Refusing update from remote peer 2ecf7a0104804308a50758f7cf0b480e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:13:47.793864 13712 raft_consensus.cc:1261] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 1 FOLLOWER]: Refusing update from remote peer 2ecf7a0104804308a50758f7cf0b480e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:13:47.794746 13789 consensus_queue.cc:1035] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30b5f0decb34297805e50b6a6987dfa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:13:47.795099 13794 consensus_queue.cc:1035] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aaee7c8de5e4d568c1796d3ceeb3e3c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:13:47.820686 13281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:13:47.821017 13281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:13:47.824803 13281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3e812afb04f48e880a2f5d0b22bebf7: Sending DeleteTablet for 3 replicas of tablet 4c8a427bac134155be47f9b5bba702f2
I20260928 05:13:47.825932 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8a427bac134155be47f9b5bba702f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44248
I20260928 05:13:47.826125 13693 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8a427bac134155be47f9b5bba702f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54308
I20260928 05:13:47.826340 13414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8a427bac134155be47f9b5bba702f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41418
I20260928 05:13:47.831744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13320
I20260928 05:13:47.840642 13813 tablet_replica.cc:330] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: stopping tablet replica
I20260928 05:13:47.841399 13812 tablet_replica.cc:330] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: stopping tablet replica
I20260928 05:13:47.841432 13813 raft_consensus.cc:2229] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:13:47.841876 13812 raft_consensus.cc:2229] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:13:47.842371 13812 raft_consensus.cc:2258] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:13:47.842386 13813 raft_consensus.cc:2258] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:13:47.844491 13812 ts_tablet_manager.cc:1905] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:13:47.846480 13813 ts_tablet_manager.cc:1905] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:13:47.855672 13812 ts_tablet_manager.cc:1918] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:13:47.856256 13812 log.cc:1196] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/4c8a427bac134155be47f9b5bba702f2
I20260928 05:13:47.856805 13812 ts_tablet_manager.cc:1939] T 4c8a427bac134155be47f9b5bba702f2 P b30b5f0decb34297805e50b6a6987dfa: Deleting consensus metadata
I20260928 05:13:47.858769 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.833035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.0.97.1:42621)
0928 05:13:47.833267 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.833275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.833315 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:13:47.833478 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.833486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.833628 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.833765 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:47.836257 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.837727 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.844569 (+ 6842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.845569 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.845729 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.849028 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:47.849048 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:47.849072 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:47.849132 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:47.854480 (+ 5348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.855392 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.855407 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.855414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.855643 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.856082 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.856090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.856096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.857035 (+ 939us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:47.857434 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.857636 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.857976 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:13:47.859067 13814 tablet_replica.cc:330] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e: stopping tablet replica
I20260928 05:13:47.859589 13475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:47.859797 13269 catalog_manager.cc:4841] TS b30b5f0decb34297805e50b6a6987dfa (127.0.97.2:40065): tablet 4c8a427bac134155be47f9b5bba702f2 (table test-table [id=1da93d208a24477abd16a5d19f9cf65a]) successfully deleted
I20260928 05:13:47.865203 13813 ts_tablet_manager.cc:1918] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:13:47.865362 13813 log.cc:1196] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/4c8a427bac134155be47f9b5bba702f2
I20260928 05:13:47.865762 13813 ts_tablet_manager.cc:1939] T 4c8a427bac134155be47f9b5bba702f2 P 8aaee7c8de5e4d568c1796d3ceeb3e3c: Deleting consensus metadata
I20260928 05:13:47.867347 13269 catalog_manager.cc:4841] TS 8aaee7c8de5e4d568c1796d3ceeb3e3c (127.0.97.3:44405): tablet 4c8a427bac134155be47f9b5bba702f2 (table test-table [id=1da93d208a24477abd16a5d19f9cf65a]) successfully deleted
I20260928 05:13:47.859532 13814 raft_consensus.cc:2229] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:13:47.957578 13814 raft_consensus.cc:2258] T 4c8a427bac134155be47f9b5bba702f2 P 2ecf7a0104804308a50758f7cf0b480e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:13:47.957772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13320
W20260928 05:13:47.974706 13269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42621 (error 108)
I20260928 05:13:47.975390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13484
I20260928 05:13:47.976460 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.975919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41440)
0928 05:13:47.976123 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:47.976255 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 05:13:47.976692 13269 catalog_manager.cc:4584] TS 2ecf7a0104804308a50758f7cf0b480e (127.0.97.1:42621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c8a427bac134155be47f9b5bba702f2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
I20260928 05:13:47.993273 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:47.976101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.0.97.2:40065)
0928 05:13:47.976260 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:47.976265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:47.976363 (+ 98us) server_negotiation.cc:407] Connection header received
0928 05:13:47.976601 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:47.976606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:47.976695 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:47.976825 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:47.978087 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.978864 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.980127 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:47.980517 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:47.980642 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:47.987736 (+ 7094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:47.987751 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:47.987768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:47.987810 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:47.990463 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.991213 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.991221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.991224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.991352 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:47.991720 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:47.991726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:47.991734 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:47.992080 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:47.992370 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:47.992684 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 05:13:47.992937 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:13:47.993903 13619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:48.030575 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.030021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41446)
0928 05:13:48.030252 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:48.030381 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 05:13:48.031702 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.031321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41456)
0928 05:13:48.031477 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:48.031563 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:13:48.084137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13484
I20260928 05:13:48.092275 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.091645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41460)
0928 05:13:48.091902 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:48.092023 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 05:13:48.093513 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.093041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41464)
0928 05:13:48.093243 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:48.093343 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:13:48.100991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13629
I20260928 05:13:48.118489 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.101840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.0.97.3:44405)
0928 05:13:48.102022 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:48.102029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:48.102125 (+ 96us) server_negotiation.cc:407] Connection header received
0928 05:13:48.102287 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:48.102291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:48.102354 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:48.102448 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:48.103874 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:48.104582 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:48.105983 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:48.106464 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:48.106726 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:48.110360 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:48.110403 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:48.110430 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:48.110478 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:48.115113 (+ 4635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:48.116201 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:48.116211 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:48.116215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:48.116366 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:48.116897 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:48.116901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:48.116903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:48.117419 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:48.117649 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:48.118003 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:13:48.118172 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:13:48.119251 13753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:48.174407 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.173927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41470)
0928 05:13:48.174139 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:48.174235 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 05:13:48.175436 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:48.175101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42621 (local address 127.0.0.1:41474)
0928 05:13:48.175276 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:48.175340 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 05:13:48.207721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13629
I20260928 05:13:48.224578 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13250
I20260928 05:13:48.225718 13313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:48.338076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13250
I20260928 05:13:48.358822 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 05:13:48.703704 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 05:13:48.703963 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 05:13:49.048754 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 05:13:49.049 UTC [12303] LOG: received smart shutdown request
2026-09-28 05:13:49.051 UTC [12303] LOG: background worker "logical replication launcher" (PID 12312) exited with exit code 1
2026-09-28 05:13:49.052 UTC [12307] LOG: shutting down
2026-09-28 05:13:49.205 UTC [12303] LOG: database system is shut down
2026-09-28T05:13:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51990 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:13:49 dist-test-slave-6718 krb5kdc[13828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:13:49 dist-test-slave-6718 krb5kdc[13828](info): set up 2 sockets
Sep 28 05:13:49 dist-test-slave-6718 krb5kdc[13828](info): commencing operation
krb5kdc: starting...
W20260928 05:13:50.427001 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:13:50 dist-test-slave-6718 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:13:50.641587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45835,127.0.97.61:42749,127.0.97.60:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:50.969264 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:50.969617 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:50.969779 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:50.977039 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:50.977185 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:50.977294 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:50.977397 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:50.977487 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:50.987211 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45835,127.0.97.61:42749,127.0.97.60:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:50.988576 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:50.989908 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:51.001549 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:51.001549 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:51.001585 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:51.002568 13841 server_base.cc:1034] running on GCE node
I20260928 05:13:51.003290 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:51.004547 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:51.005796 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790572431005749 us; error 51 us; skew 500 ppm
I20260928 05:13:51.006155 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:51.009203 13841 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:13:51.010656 13841 webserver.cc:460] Webserver started at http://127.0.97.62:44461/ using document root <none> and password file <none>
I20260928 05:13:51.011272 13841 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:51.011368 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:51.011615 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:51.014425 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "14355806d303439880c69ac454af98c7"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "1a9d6a0faa7f94e1041784919abf1158"
server_key_iv: "52d514af8e3a44876ad42bbc6d1f204c"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.015234 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "14355806d303439880c69ac454af98c7"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "1a9d6a0faa7f94e1041784919abf1158"
server_key_iv: "52d514af8e3a44876ad42bbc6d1f204c"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.020201 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:13:51.023559 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:51.024964 13841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:13:51.025107 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "14355806d303439880c69ac454af98c7"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "1a9d6a0faa7f94e1041784919abf1158"
server_key_iv: "52d514af8e3a44876ad42bbc6d1f204c"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.025290 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:51.054374 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:51.058296 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:51.058552 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:51.076807 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45835
I20260928 05:13:51.076872 13907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45835 every 8 connection(s)
I20260928 05:13:51.078159 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:13:51.081652 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:51.082967 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13841
I20260928 05:13:51.083189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:13:51.083838 388 external_mini_cluster.cc:1428] Setting key 30b740258055becb2e3daebbb0953b72
I20260928 05:13:51.088151 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260928 05:13:51.111539 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.091235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45835 (local address 127.0.0.1:39952)
0928 05:13:51.092261 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.092319 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:51.094148 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:51.095058 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:51.095063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:51.095591 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:51.095914 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.095930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.098253 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.098257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.099376 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.099387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.100740 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.100745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.100854 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.101496 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:51.101516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:51.104080 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:51.107775 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.107785 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.107792 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.108151 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.108616 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.108619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.108621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.108772 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.109710 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:51.109715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:51.109912 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 05:13:51.110100 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 05:13:51.110470 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":382,"threads_started":1}
I20260928 05:13:51.111567 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.091234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.0.97.62:45835)
0928 05:13:51.092896 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:51.092909 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:51.092962 (+ 53us) server_negotiation.cc:407] Connection header received
0928 05:13:51.094511 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:51.094522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:51.094828 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:51.095032 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:51.096109 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.098077 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.099516 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.100510 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.100733 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.104313 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:51.104320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:51.104330 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:51.104356 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:51.107633 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.108310 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.108322 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.108328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.108506 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.108889 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.108897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.108902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.109584 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:51.109787 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:51.110303 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 05:13:51.110678 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":1443,"spinlock_wait_cycles":12672,"thread_start_us":1243,"threads_started":1}
I20260928 05:13:51.114522 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:13:51.115754 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.115247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:35986)
0928 05:13:51.115460 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.115587 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 05:13:51.116042 13859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
I20260928 05:13:51.117727 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.117269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36002)
0928 05:13:51.117451 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.117552 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:13:51.117945 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42749: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 05:13:51.134419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42749
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45835,127.0.97.61:42749,127.0.97.60:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:13:51.167300 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } attempt: 1
I20260928 05:13:51.168529 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.168013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36016)
0928 05:13:51.168232 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.168349 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:13:51.169559 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.169166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36030)
0928 05:13:51.169323 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.169405 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:13:51.169751 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42749: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
I20260928 05:13:51.238030 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } attempt: 2
I20260928 05:13:51.239385 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.238815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36042)
0928 05:13:51.239099 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.239211 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 05:13:51.240419 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.240016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36048)
0928 05:13:51.240174 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.240254 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 05:13:51.240643 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42749: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
I20260928 05:13:51.331918 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } attempt: 3
I20260928 05:13:51.333103 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.332583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36056)
0928 05:13:51.332815 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.332927 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 05:13:51.334189 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.333781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36060)
0928 05:13:51.333940 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.334023 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:13:51.334374 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42749: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
W20260928 05:13:51.459316 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:51.459625 13915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:51.459725 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:51.466917 13915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:51.467073 13915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:51.467146 13915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:51.467279 13915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:51.467366 13915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:51.477140 13915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45835,127.0.97.61:42749,127.0.97.60:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:51.477607 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } attempt: 4
I20260928 05:13:51.478752 13915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:51.478885 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.478221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36068)
0928 05:13:51.478548 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.478714 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260928 05:13:51.479928 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.479509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36074)
0928 05:13:51.479688 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.479791 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 05:13:51.480139 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42749: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42749: connect: Connection refused (error 111)
I20260928 05:13:51.480379 13915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:51.491755 13920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:51.491878 13923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:51.491849 13921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:51.492761 13915 server_base.cc:1034] running on GCE node
I20260928 05:13:51.493495 13915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:51.494771 13915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:51.495968 13915 hybrid_clock.cc:648] HybridClock initialized: now 1790572431495932 us; error 33 us; skew 500 ppm
I20260928 05:13:51.496260 13915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:51.501942 13915 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 05:13:51.503415 13915 webserver.cc:460] Webserver started at http://127.0.97.61:33263/ using document root <none> and password file <none>
I20260928 05:13:51.504024 13915 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:51.504122 13915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:51.504364 13915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:51.507159 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data/instance:
uuid: "901f9f28ed57455191fcdbeba0648c4d"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "d3c19069002724cb18e707a65bace955"
server_key_iv: "56452fa112b2e3502a4a857ccca3233a"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.507874 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal/instance:
uuid: "901f9f28ed57455191fcdbeba0648c4d"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "d3c19069002724cb18e707a65bace955"
server_key_iv: "56452fa112b2e3502a4a857ccca3233a"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.512753 13915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:13:51.516139 13930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:51.517727 13915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:13:51.517930 13915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal
uuid: "901f9f28ed57455191fcdbeba0648c4d"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "d3c19069002724cb18e707a65bace955"
server_key_iv: "56452fa112b2e3502a4a857ccca3233a"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.518162 13915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:51.536907 13915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:51.540792 13915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:51.541059 13915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:51.558992 13915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42749
I20260928 05:13:51.559027 13981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42749 every 8 connection(s)
I20260928 05:13:51.560341 13915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:13:51.563896 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:51.564467 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13915
I20260928 05:13:51.564649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-1/wal/instance
I20260928 05:13:51.565184 388 external_mini_cluster.cc:1428] Setting key f9ebba432a0d0ee132cd2d8c7186c37f
I20260928 05:13:51.573669 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:51.595258 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.576169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.97.62:45835)
0928 05:13:51.576362 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:51.576366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:51.577596 (+ 1230us) server_negotiation.cc:407] Connection header received
0928 05:13:51.579497 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:51.579502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:51.579597 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:51.579710 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:51.581099 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.581779 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.583446 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.583868 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.584005 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.589648 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:51.589653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:51.589657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:51.589692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:51.592277 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.593009 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.593013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.593016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.593127 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.593559 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.593564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.593567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.593846 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:51.594015 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:51.594708 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 05:13:51.594928 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:13:51.596524 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.576721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45835 (local address 127.0.0.1:39960)
0928 05:13:51.577439 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.577481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:51.579237 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:51.579928 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:51.579936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:51.580419 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:51.580938 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.580952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.581943 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.581947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.583296 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.583308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.584086 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.584092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.584305 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.584960 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:51.584981 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:51.589413 (+ 4432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:51.592429 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.592438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.592453 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.592873 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.593250 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.593255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.593259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.593437 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.593977 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:51.593984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:51.594328 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 05:13:51.594609 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 05:13:51.595108 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":299,"threads_started":1}
I20260928 05:13:51.598186 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:13:51.614013 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.598985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36082)
0928 05:13:51.599195 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.599212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:51.599330 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:51.600298 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:51.600306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:51.600826 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:51.601033 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.601047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.602347 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.602356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.603970 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.603987 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.604871 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.604884 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.605142 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.605748 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:51.605778 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:51.608242 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:51.611629 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.611633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.611635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.612040 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.612382 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.612387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.612391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.612524 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.613215 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:51.613218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:51.613347 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 05:13:51.613456 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 05:13:51.613756 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 05:13:51.614182 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.599199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.0.97.61:42749)
0928 05:13:51.599902 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:51.599908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:51.599931 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:13:51.599969 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:51.599974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:51.600131 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:51.600242 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:51.601186 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.602187 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.604128 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.604705 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.604804 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.608481 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:51.608495 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:51.608512 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:51.608558 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:51.611517 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.612154 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.612161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.612163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.612275 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.612619 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.612622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.612624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.613103 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:51.613276 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:51.613625 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 05:13:51.613855 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":559,"thread_start_us":422,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 05:13:51.616531 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:13:51.617468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38441
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45835,127.0.97.61:42749,127.0.97.60:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:13:51.617764 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.617307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41340)
0928 05:13:51.617457 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.617563 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 05:13:51.618034 13932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.619653 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.619296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41344)
0928 05:13:51.619462 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.619545 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:13:51.619890 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.669180 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 1
I20260928 05:13:51.670387 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.669850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41354)
0928 05:13:51.670074 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.670201 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 05:13:51.671419 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.671035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41364)
0928 05:13:51.671192 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.671290 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 05:13:51.671648 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.739884 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 2
I20260928 05:13:51.741225 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.740595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41380)
0928 05:13:51.740860 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.741001 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 05:13:51.742285 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.741909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41396)
0928 05:13:51.742073 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.742152 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:13:51.742568 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.779428 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } attempt: 5
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:13:51.794100 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.780209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36084)
0928 05:13:51.780430 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.780461 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:51.780580 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:51.780938 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:51.780941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:51.781316 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:51.781494 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.781503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.783144 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.783148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.784041 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:51.784054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.785019 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.785023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:51.785182 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.785834 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:51.785852 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:51.788072 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:51.791846 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.791851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.791853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.792213 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.792581 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:51.792585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:51.792587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:51.792712 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:51.793373 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:51.793376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:51.793501 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 05:13:51.793618 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 05:13:51.793818 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:13:51.794507 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.780343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.0.97.61:42749)
0928 05:13:51.780555 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:51.780562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:51.780586 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:13:51.780695 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:51.780700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:51.780786 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:51.780904 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:51.781673 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.782951 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.784194 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:51.784878 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:51.785009 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:51.788274 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:51.788281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:51.788286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:51.788325 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:51.791696 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.792329 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.792335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.792338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.792477 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:51.792833 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:51.792838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:51.792841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:51.793247 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:51.793420 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:51.793795 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 05:13:51.794102 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:13:51.795612 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:13:51.796775 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.796273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41406)
0928 05:13:51.796495 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.796597 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 05:13:51.797703 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.797354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41408)
0928 05:13:51.797495 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.797592 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 05:13:51.797912 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.833829 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 3
I20260928 05:13:51.835161 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.834518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41424)
0928 05:13:51.834836 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.834950 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 05:13:51.836340 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.835838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41436)
0928 05:13:51.836096 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.836193 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260928 05:13:51.836566 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.849143 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 1
I20260928 05:13:51.850400 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.849829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41450)
0928 05:13:51.850059 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.850180 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 05:13:51.851486 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.851085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41462)
0928 05:13:51.851252 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.851348 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:13:51.851711 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:51.919952 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 2
I20260928 05:13:51.921218 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.920643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41478)
0928 05:13:51.920896 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.921008 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 05:13:51.922209 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.921856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41480)
0928 05:13:51.922008 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.922084 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:13:51.922441 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
W20260928 05:13:51.943795 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:51.944072 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:51.944159 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:51.951571 13989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:51.951699 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:51.951747 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:51.951795 13989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:51.951850 13989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:51.961586 13989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45835,127.0.97.61:42749,127.0.97.60:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:51.962952 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:51.964211 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:51.975507 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:51.975584 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:51.975553 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:51.976562 13989 server_base.cc:1034] running on GCE node
I20260928 05:13:51.977444 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:51.979019 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:51.979769 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 4
I20260928 05:13:51.980290 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790572431980260 us; error 69 us; skew 500 ppm
I20260928 05:13:51.980700 13989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:13:51.980877 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.980343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41488)
0928 05:13:51.980591 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.980667 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 05:13:51.981887 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:51.981563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41494)
0928 05:13:51.981713 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:51.981790 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 05:13:51.982111 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Sep 28 05:13:51 dist-test-slave-6718 krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:51.984135 13989 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 05:13:51.985750 13989 webserver.cc:460] Webserver started at http://127.0.97.60:37665/ using document root <none> and password file <none>
I20260928 05:13:51.986411 13989 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:51.986507 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:51.986857 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:51.989742 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data/instance:
uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "bc9bb04409f9b79f087442f7db76fbd3"
server_key_iv: "53bdc389df7016edeac649024d971936"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.990474 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal/instance:
uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "bc9bb04409f9b79f087442f7db76fbd3"
server_key_iv: "53bdc389df7016edeac649024d971936"
server_key_version: "encryptionkey@0"
I20260928 05:13:51.995560 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:13:51.998966 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:52.000269 13989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:13:52.000396 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal
uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb"
format_stamp: "Formatted at 2026-09-28 05:13:51 on dist-test-slave-6718"
server_key: "bc9bb04409f9b79f087442f7db76fbd3"
server_key_iv: "53bdc389df7016edeac649024d971936"
server_key_version: "encryptionkey@0"
I20260928 05:13:52.000548 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:52.015769 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:52.020040 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:52.020280 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:52.028644 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 3
I20260928 05:13:52.029903 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.029334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41498)
0928 05:13:52.029606 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.029728 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:13:52.031250 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.030764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41504)
0928 05:13:52.031002 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.031107 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 05:13:52.031500 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38441: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38441: connect: Connection refused (error 111)
I20260928 05:13:52.039201 13989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38441
I20260928 05:13:52.039230 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38441 every 8 connection(s)
I20260928 05:13:52.040601 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:13:52.044265 14056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:52.048395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 13989
I20260928 05:13:52.048583 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790572228671169-388-0/minicluster-data/master-2/wal/instance
I20260928 05:13:52.048990 388 external_mini_cluster.cc:1428] Setting key 96b19a6e23d39db5225e68ddf15cd1f9
I20260928 05:13:52.051141 14056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:13:52.068045 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.053618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.0.97.62:45835)
0928 05:13:52.053828 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.053835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.054511 (+ 676us) server_negotiation.cc:407] Connection header received
0928 05:13:52.054796 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.054808 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.054867 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.054974 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.056203 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.056866 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.058273 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.058767 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.058896 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.062237 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.062243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.062249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.062286 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.064966 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.065712 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.065720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.065724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.065859 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.066314 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.066321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.066324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.066648 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.066823 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.067469 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.067698 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:13:52.069339 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.050841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.0.97.60:38441)
0928 05:13:52.051618 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.051628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.051674 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:13:52.053392 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.053424 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.053903 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.054281 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.055712 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.056963 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.058313 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.058879 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.059072 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.063415 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.063423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.063434 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.063460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.066185 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.066912 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.066918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.066920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.067037 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.067433 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.067437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.067439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.067755 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.068036 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.068500 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.068745 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":573,"thread_start_us":380,"threads_started":1}
I20260928 05:13:52.069360 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.053595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45835 (local address 127.0.0.1:39964)
0928 05:13:52.054378 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.054427 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:52.054653 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:52.055038 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:52.055045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:52.055606 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:52.056038 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.056052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.057031 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.057035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.058123 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.058134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.059006 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.059014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.059208 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.059792 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:52.059812 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:52.062043 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:52.065112 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.065122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.065133 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.065584 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.065969 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.065974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.065982 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.066201 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.066786 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:52.066796 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:52.067110 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 05:13:52.067376 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 05:13:52.067824 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":44,"spinlock_wait_cycles":14592,"thread_start_us":379,"threads_started":1}
I20260928 05:13:52.070837 14056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:52.086724 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.071609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42749 (local address 127.0.0.1:36090)
0928 05:13:52.071784 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.071798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:52.071870 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:52.072580 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:52.072585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:52.073019 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:52.073220 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.073231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.074656 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.074661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.075836 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.075847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.076809 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.076815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.077014 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.077630 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:52.077651 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:52.080251 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:52.084047 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.084054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.084058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.084563 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.084982 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.084987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.084991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.085180 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.085729 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:52.085735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:52.085890 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 05:13:52.086052 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 05:13:52.086352 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 05:13:52.086764 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.071895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.0.97.61:42749)
0928 05:13:52.072115 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.072120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.072139 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:13:52.072195 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.072200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.072277 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.072371 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.073395 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.074481 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.075991 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.076640 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.076765 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.080443 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.080449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.080454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.080495 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.083911 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.084685 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.084690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.084694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.084856 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.085300 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.085305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.085308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.085603 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.085762 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.086171 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.086363 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:13:52.088120 14056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:13:52.088771 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.072404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.0.97.62:45835)
0928 05:13:52.072580 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.072586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.072601 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:13:52.072645 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.072653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.072717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.072821 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.074315 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.075369 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.076954 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.077520 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.077643 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.082087 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.082092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.082095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.082123 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.085435 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.086226 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.086231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.086236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.086389 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.087210 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.087216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.087218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.087553 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.087857 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.088206 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.088421 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:13:52.102279 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.088944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41524)
0928 05:13:52.089343 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.089365 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:52.089483 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:52.089803 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:52.089808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:52.090287 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:52.090463 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.090475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.091593 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.091601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.092723 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.092735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.093488 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.093494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.093736 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.094290 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:52.094313 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:52.096427 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:52.099868 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.099873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.099877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.100264 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.100653 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.100659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.100664 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.100824 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.101428 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:52.101432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:52.101544 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 05:13:52.101703 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 05:13:52.102002 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20260928 05:13:52.102625 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.088914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.0.97.60:38441)
0928 05:13:52.089085 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.089090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.089442 (+ 352us) server_negotiation.cc:407] Connection header received
0928 05:13:52.089594 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.089598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.089650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.089760 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.090623 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.091460 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.092852 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.093345 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.093431 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.096624 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.096630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.096638 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.096675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.099739 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.100376 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.100381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.100384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.100532 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.100940 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.100945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.100948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.101313 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.101460 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.101829 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.102178 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":60}
I20260928 05:13:52.106819 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.090434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.0.97.61:42749)
0928 05:13:52.090621 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.090626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.090643 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:13:52.090781 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.090785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.090850 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.090949 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.092203 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.092952 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.094612 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.095016 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.095108 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.100043 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.100048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.100051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.100084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.103409 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.104330 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.104334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.104336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.104542 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.105035 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.105041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.105049 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.105375 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.105577 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.105945 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.106197 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 05:13:52.108256 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: Bootstrap starting.
I20260928 05:13:52.112723 14056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:52.113894 14056 log.cc:824] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:52.114660 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:13:52.116447 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: No bootstrap required, opened a new log
I20260928 05:13:52.126751 14056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.130586 14056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:52.130818 14056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a031c5818aa4ddbb4f1d6e7872a78bb, State: Initialized, Role: FOLLOWER
I20260928 05:13:52.131691 14056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.132752 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.133260 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.136467 14056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:13:52.138770 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.119294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.0.97.62:45835)
0928 05:13:52.119481 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.119487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.120480 (+ 993us) server_negotiation.cc:407] Connection header received
0928 05:13:52.120752 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.120759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.120841 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.120914 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.123301 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.124420 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.126665 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.127292 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.127414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.131656 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.131666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.131674 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.131726 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.134977 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.135951 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.135956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.135959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.136124 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.136590 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.136596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.136603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.136995 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.137238 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.137877 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.138203 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:13:52.147572 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.119514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.97.61:42749)
0928 05:13:52.119705 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.119710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.127870 (+ 8160us) server_negotiation.cc:407] Connection header received
0928 05:13:52.128076 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.128084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.128176 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.128253 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.132329 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.133446 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.135480 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.136135 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.136238 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.139788 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.139795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.139802 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.139849 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.144025 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.144872 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.144878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.144884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.145052 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.145542 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.145547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.145550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.145972 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.146225 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.146766 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.147246 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:13:52.149667 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.121389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.0.97.60:38441)
0928 05:13:52.123462 (+ 2073us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.123471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.131303 (+ 7832us) server_negotiation.cc:407] Connection header received
0928 05:13:52.131483 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.131489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.131594 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.131697 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.133188 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.134219 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.136340 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.137178 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.137297 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.140951 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.140961 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.140968 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.141017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.145770 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.146614 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.146619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.146622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.146819 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.147322 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.147329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.147334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.147762 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.147988 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.149046 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.149259 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1899}
W20260928 05:13:52.157398 14080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 05:13:52.149031 13989 master_runner.cc:186] Error getting master registration for 127.0.97.62:45835: OK, instance_id { permanent_uuid: "14355806d303439880c69ac454af98c7" instance_seqno: 1790572431071478 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 05:13:52.157609 14080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:13:52.157753 13989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:13:52.208725 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 4
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:13:52.223246 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.209323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41542)
0928 05:13:52.209527 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.209548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:52.209664 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:52.210041 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:52.210047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:52.210460 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:52.210710 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.210720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.211762 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.211768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.212648 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.212659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.213424 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.213429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.213621 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.214246 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:52.214263 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:52.216754 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:52.220844 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.220849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.220851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.221250 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.221650 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.221653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.221655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.221835 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.222434 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:52.222437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:52.222591 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 05:13:52.222709 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 05:13:52.222974 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 05:13:52.223443 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.209407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.0.97.60:38441)
0928 05:13:52.209561 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.209569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.209591 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:13:52.209813 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.209825 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.209904 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.210017 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.210860 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.211605 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.212795 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.213263 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.213415 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.217066 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.217074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.217081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.217132 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.220709 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.221361 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.221367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.221371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.221538 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.221947 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.221971 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.221974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.222321 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.222478 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.222827 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.223090 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:13:52.229589 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: Bootstrap starting.
I20260928 05:13:52.233634 13908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:52.234968 13908 log.cc:824] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:52.238008 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: No bootstrap required, opened a new log
I20260928 05:13:52.249042 13908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.249466 13908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:52.249588 13908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14355806d303439880c69ac454af98c7, State: Initialized, Role: FOLLOWER
I20260928 05:13:52.250275 13908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.251210 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.251546 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.252625 13908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:13:52.263741 14094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:13:52.263955 14094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:13:52.261305 13841 master_runner.cc:186] Error getting master registration for 127.0.97.61:42749: OK, instance_id { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" instance_seqno: 1790572431553794 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 05:13:52.272279 14010 tablet.cc:2369] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:52.272468 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:13:52.281353 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } attempt: 5
Sep 28 05:13:52 dist-test-slave-6718 krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:13:52.299897 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.282026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38441 (local address 127.0.0.1:41558)
0928 05:13:52.282385 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:52.282405 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:52.282524 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:52.284301 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:52.284307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:52.284846 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:52.285049 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.285061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.286526 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.286562 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.288027 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:52.288042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.289050 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.289056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:52.289249 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.289939 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:52.289963 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:52.292819 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:52.296961 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.296966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.296971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.297469 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.297919 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:52.297925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:52.297929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:52.298093 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:52.298843 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:52.298849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:52.299011 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 05:13:52.299157 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 05:13:52.299442 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260928 05:13:52.301008 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:52.282678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.0.97.60:38441)
0928 05:13:52.283110 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:52.283116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:52.283135 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:13:52.283964 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:52.283970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:52.284044 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:52.284130 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:52.285210 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.286360 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.288176 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:52.288871 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:52.289005 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:52.293025 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:52.293032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:52.293038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:52.293081 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:52.296819 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.297633 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.297639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.297643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.297802 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:52.298224 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:52.298229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:52.298232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:52.298629 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:52.298779 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:52.300369 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0928 05:13:52.300633 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260928 05:13:52.307297 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: Bootstrap starting.
W20260928 05:13:52.311229 13862 tablet.cc:2369] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:52.311322 13982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:52.312562 13982 log.cc:824] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:52.315778 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: No bootstrap required, opened a new log
I20260928 05:13:52.326629 13982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.327054 13982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:52.327198 13982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901f9f28ed57455191fcdbeba0648c4d, State: Initialized, Role: FOLLOWER
I20260928 05:13:52.327838 13982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.328846 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.329366 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.330183 13982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:13:52.351327 13915 master_runner.cc:263] No leader master found from master 901f9f28ed57455191fcdbeba0648c4d
I20260928 05:13:52.351502 13915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 05:13:52.357380 14108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:13:52.357545 14108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:13:52.540831 14069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:13:52.541179 14069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.543141 14069 leader_election.cc:288] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14355806d303439880c69ac454af98c7 (127.0.97.62:45835), 901f9f28ed57455191fcdbeba0648c4d (127.0.97.61:42749)
W20260928 05:13:52.543548 13936 tablet.cc:2369] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:52.544716 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14355806d303439880c69ac454af98c7" is_pre_election: true
I20260928 05:13:52.545337 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a031c5818aa4ddbb4f1d6e7872a78bb in term 0.
I20260928 05:13:52.546438 14005 leader_election.cc:302] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a031c5818aa4ddbb4f1d6e7872a78bb, 14355806d303439880c69ac454af98c7; no voters:
I20260928 05:13:52.547004 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:13:52.547190 14069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:13:52.547307 14069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:52.551144 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901f9f28ed57455191fcdbeba0648c4d" is_pre_election: true
I20260928 05:13:52.551772 13957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a031c5818aa4ddbb4f1d6e7872a78bb in term 0.
I20260928 05:13:52.557638 14069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.559428 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901f9f28ed57455191fcdbeba0648c4d"
I20260928 05:13:52.559661 13957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:52.561374 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14355806d303439880c69ac454af98c7"
I20260928 05:13:52.561602 13883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:52.563432 13957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a031c5818aa4ddbb4f1d6e7872a78bb in term 1.
I20260928 05:13:52.563625 14069 leader_election.cc:288] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [CANDIDATE]: Term 1 election: Requested vote from peers 14355806d303439880c69ac454af98c7 (127.0.97.62:45835), 901f9f28ed57455191fcdbeba0648c4d (127.0.97.61:42749)
I20260928 05:13:52.564378 14007 leader_election.cc:302] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a031c5818aa4ddbb4f1d6e7872a78bb, 901f9f28ed57455191fcdbeba0648c4d; no voters:
I20260928 05:13:52.564934 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:13:52.566811 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a031c5818aa4ddbb4f1d6e7872a78bb in term 1.
I20260928 05:13:52.573593 14069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [term 1 LEADER]: Becoming Leader. State: Replica: 0a031c5818aa4ddbb4f1d6e7872a78bb, State: Running, Role: LEADER
I20260928 05:13:52.574435 14069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } }
I20260928 05:13:52.586652 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a031c5818aa4ddbb4f1d6e7872a78bb. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.587239 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:52.599242 14112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:13:52.601557 14112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:13:52.618451 13957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [term 1 FOLLOWER]: Refusing update from remote peer 0a031c5818aa4ddbb4f1d6e7872a78bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:13:52.619752 14110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:13:52.630106 13883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [term 1 FOLLOWER]: Refusing update from remote peer 0a031c5818aa4ddbb4f1d6e7872a78bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:13:52.631373 14110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:13:52.663394 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a031c5818aa4ddbb4f1d6e7872a78bb. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.663702 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.665012 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a031c5818aa4ddbb4f1d6e7872a78bb. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.665339 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.666613 14113 mvcc.cc:205] Tried to move back new op lower bound from 7334184683990061056 to 7334184683831943168. Current Snapshot: MvccSnapshot[applied={T|T < 7334184683990061056}]
I20260928 05:13:52.677423 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.677649 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.679993 14112 catalog_manager.cc:1347] Generated new cluster ID: f69733ffffed429bab059c03ba3a2bc6
I20260928 05:13:52.680086 14112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:13:52.680405 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.680660 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:52.681023 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.681231 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:52.687093 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14355806d303439880c69ac454af98c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45835 } } peers { permanent_uuid: "901f9f28ed57455191fcdbeba0648c4d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42749 } } peers { permanent_uuid: "0a031c5818aa4ddbb4f1d6e7872a78bb" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38441 } } }
I20260928 05:13:52.687350 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:13:52.711174 14112 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:13:52.713441 14112 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:13:52.737082 14112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a031c5818aa4ddbb4f1d6e7872a78bb: Generated new TSK 0
I20260928 05:13:52.738291 14112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:13:53.265870 14094 catalog_manager.cc:1259] Loaded cluster ID: f69733ffffed429bab059c03ba3a2bc6
I20260928 05:13:53.265969 14094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: loading cluster ID for follower catalog manager: success
I20260928 05:13:53.268287 14094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: acquiring CA information for follower catalog manager: success
I20260928 05:13:53.269903 14094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14355806d303439880c69ac454af98c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:13:53.359565 14108 catalog_manager.cc:1259] Loaded cluster ID: f69733ffffed429bab059c03ba3a2bc6
I20260928 05:13:53.359724 14108 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: loading cluster ID for follower catalog manager: success
I20260928 05:13:53.362339 14108 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: acquiring CA information for follower catalog manager: success
I20260928 05:13:53.364266 14108 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 901f9f28ed57455191fcdbeba0648c4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:13:53.836577 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13841
I20260928 05:13:53.838235 13903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:53.949931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13841
I20260928 05:13:53.970001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13915
I20260928 05:13:53.971113 13977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:54.076120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13915
I20260928 05:13:54.096053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 13989
I20260928 05:13:54.097064 14051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:54.208200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 13989
2026-09-28T05:13:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4871 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:13:54 dist-test-slave-6718 krb5kdc[14132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:13:54 dist-test-slave-6718 krb5kdc[14132](info): set up 2 sockets
Sep 28 05:13:54 dist-test-slave-6718 krb5kdc[14132](info): commencing operation
krb5kdc: starting...
W20260928 05:13:55.314975 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:13:55 dist-test-slave-6718 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:13:55.541523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32839
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:55.865545 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:55.865875 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:55.866052 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:55.873325 14145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:55.873471 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:55.873589 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:55.873685 14145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:55.873772 14145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:55.883807 14145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:55.885064 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:55.886363 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:55.897823 14151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:55.897781 14153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:55.897863 14150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:55.898859 14145 server_base.cc:1034] running on GCE node
I20260928 05:13:55.899605 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:55.900944 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:55.902148 14145 hybrid_clock.cc:648] HybridClock initialized: now 1790572435902108 us; error 26 us; skew 500 ppm
I20260928 05:13:55.902500 14145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:55 dist-test-slave-6718 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:55.905604 14145 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:13:55.907143 14145 webserver.cc:460] Webserver started at http://127.0.97.62:44411/ using document root <none> and password file <none>
I20260928 05:13:55.907829 14145 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:55.907949 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:55.908273 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:55.911283 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "aa693c09b432463abea6625fc9549408"
format_stamp: "Formatted at 2026-09-28 05:13:55 on dist-test-slave-6718"
server_key: "482ff931690cc6518a8a57226701d829"
server_key_iv: "ec727cab2a4ffbeaa26d91904e4ee61c"
server_key_version: "encryptionkey@0"
I20260928 05:13:55.912070 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa693c09b432463abea6625fc9549408"
format_stamp: "Formatted at 2026-09-28 05:13:55 on dist-test-slave-6718"
server_key: "482ff931690cc6518a8a57226701d829"
server_key_iv: "ec727cab2a4ffbeaa26d91904e4ee61c"
server_key_version: "encryptionkey@0"
I20260928 05:13:55.917160 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:13:55.921994 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:55.923355 14145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 05:13:55.923540 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "aa693c09b432463abea6625fc9549408"
format_stamp: "Formatted at 2026-09-28 05:13:55 on dist-test-slave-6718"
server_key: "482ff931690cc6518a8a57226701d829"
server_key_iv: "ec727cab2a4ffbeaa26d91904e4ee61c"
server_key_version: "encryptionkey@0"
I20260928 05:13:55.923731 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:55.939982 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:55.944192 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:55.944490 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:55.963056 14145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32839
I20260928 05:13:55.963090 14211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32839 every 8 connection(s)
I20260928 05:13:55.964432 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:13:55.968398 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:55.973248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14145
I20260928 05:13:55.973474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:13:55.974362 388 external_mini_cluster.cc:1428] Setting key 6205d31b4326ec7ba0a07d084d2bf203
I20260928 05:13:55.981195 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408: Bootstrap starting.
I20260928 05:13:55.985184 14212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:55.986392 14212 log.cc:824] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:55.989871 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408: No bootstrap required, opened a new log
Sep 28 05:13:55 dist-test-slave-6718 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:56.000379 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:55.976235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.0.97.62:32839)
0928 05:13:55.977017 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:55.977029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:55.977068 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:13:55.979353 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:55.979375 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:55.979752 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:55.980059 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:55.982070 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:55.983536 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:55.985647 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:55.986302 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:55.986430 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:55.991486 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:55.991497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:55.991516 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:55.991565 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:55.995593 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:55.996313 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:55.996323 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:55.996333 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:55.996554 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:55.996887 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:55.996893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:55.996903 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:55.997664 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:55.997948 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:55.998348 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 05:13:55.998861 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":376,"threads_started":1}
I20260928 05:13:56.001035 14212 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa693c09b432463abea6625fc9549408" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260928 05:13:56.001633 14212 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:56.001770 14212 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa693c09b432463abea6625fc9549408, State: Initialized, Role: FOLLOWER
I20260928 05:13:56.002410 14212 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa693c09b432463abea6625fc9549408" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260928 05:13:56.002749 14212 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:13:56.002907 14212 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:13:56.003109 14212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:56.004678 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:13:56.005718 14212 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa693c09b432463abea6625fc9549408" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260928 05:13:56.006213 14212 leader_election.cc:302] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa693c09b432463abea6625fc9549408; no voters:
I20260928 05:13:56.006881 14212 leader_election.cc:288] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:13:56.006996 14217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:13:56.008430 14217 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [term 1 LEADER]: Becoming Leader. State: Replica: aa693c09b432463abea6625fc9549408, State: Running, Role: LEADER
I20260928 05:13:56.008925 14217 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa693c09b432463abea6625fc9549408" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260928 05:13:56.009182 14212 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:13:56.011668 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa693c09b432463abea6625fc9549408. Latest consensus state: current_term: 1 leader_uuid: "aa693c09b432463abea6625fc9549408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa693c09b432463abea6625fc9549408" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } } }
I20260928 05:13:56.012115 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:56.012992 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa693c09b432463abea6625fc9549408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa693c09b432463abea6625fc9549408" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } } }
I20260928 05:13:56.013360 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:56.013355 14224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:13:56.018690 14224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:13:56.027601 14224 catalog_manager.cc:1347] Generated new cluster ID: 8ced2ffa498e43c2bea1d8c4e51b959b
I20260928 05:13:56.027719 14224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:13:56.040591 14224 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:13:56.042495 14224 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:13:56.080366 14224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa693c09b432463abea6625fc9549408: Generated new TSK 0
I20260928 05:13:56.081427 14224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:13:56.119760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14145
I20260928 05:13:56.120926 14207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:56.220271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14145
2026-09-28T05:13:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2007 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:13:56 dist-test-slave-6718 krb5kdc[14241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:13:56 dist-test-slave-6718 krb5kdc[14241](info): set up 2 sockets
Sep 28 05:13:56 dist-test-slave-6718 krb5kdc[14241](info): commencing operation
krb5kdc: starting...
W20260928 05:13:57.323177 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:13:57 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:13:57.550133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35525
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:57.871836 14254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:57.872109 14254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:57.872308 14254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:57.879381 14254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:13:57.879477 14254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:57.879590 14254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:57.879710 14254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:13:57.879796 14254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:13:57.889542 14254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:57.890887 14254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:57.892217 14254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:57.903753 14259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:57.903673 14262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:57.903715 14260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:57.904563 14254 server_base.cc:1034] running on GCE node
I20260928 05:13:57.905354 14254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:57.906839 14254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:57.908041 14254 hybrid_clock.cc:648] HybridClock initialized: now 1790572437907995 us; error 31 us; skew 500 ppm
I20260928 05:13:57.908393 14254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:57 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:57.911547 14254 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:13:57.913081 14254 webserver.cc:460] Webserver started at http://127.0.97.62:34587/ using document root <none> and password file <none>
I20260928 05:13:57.913782 14254 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:57.913930 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:57.914242 14254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:57.917287 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "ec0aaf2b787f4b59b05d2f2078542ba1"
format_stamp: "Formatted at 2026-09-28 05:13:57 on dist-test-slave-6718"
server_key: "51bfaf8659d21d40d0dcf5814235e4c8"
server_key_iv: "cc3ed5e701d457bb4e0ab2a5108c6e3c"
server_key_version: "encryptionkey@0"
I20260928 05:13:57.918067 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "ec0aaf2b787f4b59b05d2f2078542ba1"
format_stamp: "Formatted at 2026-09-28 05:13:57 on dist-test-slave-6718"
server_key: "51bfaf8659d21d40d0dcf5814235e4c8"
server_key_iv: "cc3ed5e701d457bb4e0ab2a5108c6e3c"
server_key_version: "encryptionkey@0"
I20260928 05:13:57.923358 14254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 05:13:57.927591 14269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:57.929427 14254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 05:13:57.929631 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "ec0aaf2b787f4b59b05d2f2078542ba1"
format_stamp: "Formatted at 2026-09-28 05:13:57 on dist-test-slave-6718"
server_key: "51bfaf8659d21d40d0dcf5814235e4c8"
server_key_iv: "cc3ed5e701d457bb4e0ab2a5108c6e3c"
server_key_version: "encryptionkey@0"
I20260928 05:13:57.929909 14254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:57.970829 14254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:57.975183 14254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:57.975497 14254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:57.994069 14254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35525
I20260928 05:13:57.994153 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35525 every 8 connection(s)
I20260928 05:13:57.995517 14254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:13:57.999246 14321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:58.001883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14254
I20260928 05:13:58.002094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:13:58.002919 388 external_mini_cluster.cc:1428] Setting key 7b9585ac73f8376afaf6dfab681fcee2
I20260928 05:13:58.011842 14321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap starting.
I20260928 05:13:58.015663 14321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:58.017036 14321 log.cc:824] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:13:58 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:58.020680 14321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: No bootstrap required, opened a new log
I20260928 05:13:58.028053 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:58.005074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.0.97.62:35525)
0928 05:13:58.005898 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:58.005909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:58.005951 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:13:58.007650 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:58.007673 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:58.008097 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:58.008453 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:58.010189 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:58.011541 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:58.013513 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:58.014241 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:58.014378 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:58.019891 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:58.019901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:58.019917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:58.019960 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:58.023585 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:58.024467 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:58.024474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:58.024482 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:58.024740 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:58.025196 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:58.025200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:58.025202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:58.025773 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:58.026005 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:58.026446 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 05:13:58.026991 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":388,"threads_started":1}
I20260928 05:13:58.030017 14321 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:13:58.030462 14321 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:58.030616 14321 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0aaf2b787f4b59b05d2f2078542ba1, State: Initialized, Role: FOLLOWER
I20260928 05:13:58.031309 14321 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:13:58.031581 14321 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:13:58.031749 14321 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:13:58.031924 14321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:58.034683 14321 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:13:58.035193 14321 leader_election.cc:302] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0aaf2b787f4b59b05d2f2078542ba1; no voters:
I20260928 05:13:58.035799 14321 leader_election.cc:288] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:13:58.035925 14326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:13:58.036655 14326 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 LEADER]: Becoming Leader. State: Replica: ec0aaf2b787f4b59b05d2f2078542ba1, State: Running, Role: LEADER
I20260928 05:13:58.037176 14326 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:13:58.037845 14321 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:13:58.040091 14327 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } } }
I20260928 05:13:58.040438 14327 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:58.040196 14328 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0aaf2b787f4b59b05d2f2078542ba1. Latest consensus state: current_term: 1 leader_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } } }
I20260928 05:13:58.040608 14328 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:13:58.042572 14331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:13:58.048795 14331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:13:58.057849 14331 catalog_manager.cc:1347] Generated new cluster ID: db8e19ca0e0c4121bb5e384be1416289
I20260928 05:13:58.057977 14331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:13:58.074298 14331 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:13:58.076121 14331 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:13:58.086588 14331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Generated new TSK 0
I20260928 05:13:58.087632 14331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:13:58.146345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35525
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:58.475167 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:58.475450 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:58.475601 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:58.483361 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:58.483485 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:58.483669 14347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:13:58.493503 14347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:58.494824 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:58.496131 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:58.508925 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:58.508989 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:58.509013 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:58.509728 14347 server_base.cc:1034] running on GCE node
I20260928 05:13:58.510341 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:58.511288 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:58.512497 14347 hybrid_clock.cc:648] HybridClock initialized: now 1790572438512459 us; error 31 us; skew 500 ppm
I20260928 05:13:58.512840 14347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:58 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:58.515990 14347 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:13:58.517550 14347 webserver.cc:460] Webserver started at http://127.0.97.1:42361/ using document root <none> and password file <none>
I20260928 05:13:58.518213 14347 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:58.518339 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:58.518718 14347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:58.521544 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "56bfc9ba215d4419a08fc3757806dc39"
format_stamp: "Formatted at 2026-09-28 05:13:58 on dist-test-slave-6718"
server_key: "8fb33f98c78c8749be4b606a7f138987"
server_key_iv: "286fa6f4b796ea9e9909ff3a8e5774f0"
server_key_version: "encryptionkey@0"
I20260928 05:13:58.522313 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "56bfc9ba215d4419a08fc3757806dc39"
format_stamp: "Formatted at 2026-09-28 05:13:58 on dist-test-slave-6718"
server_key: "8fb33f98c78c8749be4b606a7f138987"
server_key_iv: "286fa6f4b796ea9e9909ff3a8e5774f0"
server_key_version: "encryptionkey@0"
I20260928 05:13:58.527318 14347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 05:13:58.530594 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:58.531833 14347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:13:58.531998 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "56bfc9ba215d4419a08fc3757806dc39"
format_stamp: "Formatted at 2026-09-28 05:13:58 on dist-test-slave-6718"
server_key: "8fb33f98c78c8749be4b606a7f138987"
server_key_iv: "286fa6f4b796ea9e9909ff3a8e5774f0"
server_key_version: "encryptionkey@0"
I20260928 05:13:58.532193 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:58.555200 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:58.559751 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:58.560050 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:58.560979 14347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:58.562273 14347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:58.562376 14347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:58.562541 14347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:58.562615 14347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 05:13:58.596346 14347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38271
I20260928 05:13:58.596441 14474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38271 every 8 connection(s)
I20260928 05:13:58.597666 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:13:58.598070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14347
I20260928 05:13:58.598306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:13:58.599251 388 external_mini_cluster.cc:1428] Setting key a59915b2eda6ad6394614a405539a3ad
Sep 28 05:13:58 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:58.621846 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:58.600510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52949 (local address 127.0.97.62:35525)
0928 05:13:58.600977 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:58.600984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:58.602042 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 05:13:58.604276 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:58.604281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:58.604376 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:58.604506 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:58.606111 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:58.606928 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:58.608571 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:58.608992 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:58.609127 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:58.613740 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:58.613750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:58.613758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:58.613811 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:58.617773 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:58.618910 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:58.618917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:58.618923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:58.619108 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:58.619717 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:58.619723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:58.619727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:58.620069 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:58.620239 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:58.621210 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 05:13:58.621475 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260928 05:13:58.623301 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:58.601151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:52949)
0928 05:13:58.601897 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:58.601934 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:58.603989 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:58.604791 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:58.604800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:58.605338 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:58.605931 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:58.605950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:58.607097 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:58.607101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:58.608415 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:58.608428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:58.609239 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:58.609247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:58.609514 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:58.610281 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:58.610305 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:58.613407 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:58.617963 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:58.617975 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:58.617997 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:58.618756 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:58.619286 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:58.619291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:58.619296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:58.619513 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:58.620319 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:58.620333 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:58.620814 (+ 481us) client_negotiation.cc:769] Sending connection context
0928 05:13:58.621094 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 05:13:58.621675 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":328,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:13:58.625710 14475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35525
I20260928 05:13:58.626050 14475 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:58.626885 14475 heartbeater.cc:511] Master 127.0.97.62:35525 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:13:58.629912 14286 ts_manager.cc:194] Registered new tserver with Master: 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271)
I20260928 05:13:58.632249 14286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:13:58.658030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35525
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:58.986222 14481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:58.986490 14481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:58.986649 14481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:58.993955 14481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:58.994062 14481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:58.994273 14481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:13:59.004128 14481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:59.005383 14481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:59.006718 14481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:59.019484 14486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:59.019539 14489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:59.019511 14487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:59.020596 14481 server_base.cc:1034] running on GCE node
I20260928 05:13:59.021188 14481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:59.022117 14481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:59.023324 14481 hybrid_clock.cc:648] HybridClock initialized: now 1790572439023282 us; error 30 us; skew 500 ppm
I20260928 05:13:59.023645 14481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:59 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:59.026723 14481 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:13:59.028123 14481 webserver.cc:460] Webserver started at http://127.0.97.2:40327/ using document root <none> and password file <none>
I20260928 05:13:59.028765 14481 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:59.028864 14481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:59.029106 14481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:59.032380 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "e75c30d5de58454c839d8cb85630294e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "eeb60f01854f62affd4bc8e50b516d31"
server_key_iv: "63bd7e5a6443b6016c3415c388d5ab40"
server_key_version: "encryptionkey@0"
I20260928 05:13:59.033131 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e75c30d5de58454c839d8cb85630294e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "eeb60f01854f62affd4bc8e50b516d31"
server_key_iv: "63bd7e5a6443b6016c3415c388d5ab40"
server_key_version: "encryptionkey@0"
I20260928 05:13:59.038396 14481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:13:59.041685 14496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:59.043012 14481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:13:59.043187 14481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "e75c30d5de58454c839d8cb85630294e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "eeb60f01854f62affd4bc8e50b516d31"
server_key_iv: "63bd7e5a6443b6016c3415c388d5ab40"
server_key_version: "encryptionkey@0"
I20260928 05:13:59.043371 14481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:59.070753 14481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:59.074733 14481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:59.075026 14481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:59.075991 14481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:59.077332 14481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:59.077404 14481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:59.077576 14481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:59.077626 14481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:59.112046 14481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43551
I20260928 05:13:59.112129 14608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43551 every 8 connection(s)
I20260928 05:13:59.113890 14481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:13:59.121477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14481
I20260928 05:13:59.121657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:13:59.122076 388 external_mini_cluster.cc:1428] Setting key c49c252baf654885d761e2cf217b471b
Sep 28 05:13:59 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:59.141209 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.117197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53165 (local address 127.0.97.62:35525)
0928 05:13:59.118768 (+ 1571us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.118777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.119480 (+ 703us) server_negotiation.cc:407] Connection header received
0928 05:13:59.122596 (+ 3116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.122613 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.122759 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.122887 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:59.124879 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.125755 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.127931 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.128682 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.128836 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.133326 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:59.133335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:59.133340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:59.133409 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:59.137467 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.138597 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.138603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.138606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.138797 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.139333 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.139338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.139341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.139728 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:59.139842 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.140631 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.140866 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1365}
I20260928 05:13:59.142398 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.118093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:53165)
0928 05:13:59.119283 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.119342 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.122223 (+ 2881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.123160 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.123172 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.123879 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:59.124685 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.124712 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.125957 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.125961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.127776 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.127789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.128904 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.128909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.129106 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.129896 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:59.129919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:59.132917 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:59.137651 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:59.137664 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:59.137686 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:59.138367 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:59.138916 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:59.138922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:59.138926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:59.139186 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:59.139934 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:59.139940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:59.140261 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.140531 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.141101 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":590,"threads_started":1}
I20260928 05:13:59.144068 14609 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35525
I20260928 05:13:59.144425 14609 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:59.145141 14609 heartbeater.cc:511] Master 127.0.97.62:35525 requested a full tablet report, sending...
I20260928 05:13:59.146853 14286 ts_manager.cc:194] Registered new tserver with Master: e75c30d5de58454c839d8cb85630294e (127.0.97.2:43551)
I20260928 05:13:59.147864 14286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53165
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:13:59.177614 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35525
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:13:59.498690 14614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:13:59.499226 14614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:13:59.499410 14614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:13:59.506467 14614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:13:59.506626 14614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:13:59.506817 14614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:13:59.516343 14614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:13:59.517741 14614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:13:59.519089 14614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:13:59.532244 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:59.532300 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:13:59.532266 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:13:59.533274 14614 server_base.cc:1034] running on GCE node
I20260928 05:13:59.533891 14614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:13:59.534909 14614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:13:59.536139 14614 hybrid_clock.cc:648] HybridClock initialized: now 1790572439536084 us; error 61 us; skew 500 ppm
I20260928 05:13:59.536487 14614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:13:59 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:13:59.539796 14614 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:13:59.541246 14614 webserver.cc:460] Webserver started at http://127.0.97.3:40683/ using document root <none> and password file <none>
I20260928 05:13:59.541872 14614 fs_manager.cc:359] Metadata directory not provided
I20260928 05:13:59.541970 14614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:13:59.542212 14614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:13:59.545061 14614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "012d88fd608b83b5bbfdb3d9f392986d"
server_key_iv: "7f33ee2ed15e2a8ec0455ba158e5ce22"
server_key_version: "encryptionkey@0"
I20260928 05:13:59.545766 14614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "012d88fd608b83b5bbfdb3d9f392986d"
server_key_iv: "7f33ee2ed15e2a8ec0455ba158e5ce22"
server_key_version: "encryptionkey@0"
I20260928 05:13:59.550928 14614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 05:13:59.554280 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:59.555677 14614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:13:59.555845 14614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "012d88fd608b83b5bbfdb3d9f392986d"
server_key_iv: "7f33ee2ed15e2a8ec0455ba158e5ce22"
server_key_version: "encryptionkey@0"
I20260928 05:13:59.556051 14614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:13:59.579034 14614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:13:59.583061 14614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:13:59.583350 14614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:13:59.584365 14614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:13:59.585798 14614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:13:59.585875 14614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:59.586011 14614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:13:59.586067 14614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:13:59.621508 14614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36475
I20260928 05:13:59.621554 14741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36475 every 8 connection(s)
I20260928 05:13:59.622946 14614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:13:59.629700 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14614
I20260928 05:13:59.629885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:13:59.630301 388 external_mini_cluster.cc:1428] Setting key 2b07a2d74aa1a99f91d799f3d9b8b247
I20260928 05:13:59.635670 14475 heartbeater.cc:503] Master 127.0.97.62:35525 was elected leader, sending a full tablet report...
Sep 28 05:13:59 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:13:59.643368 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.625458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38117 (local address 127.0.97.62:35525)
0928 05:13:59.625682 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.625686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.626974 (+ 1288us) server_negotiation.cc:407] Connection header received
0928 05:13:59.628698 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.628703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.628766 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.628859 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:59.630546 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.631592 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.633707 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.634085 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.634208 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.638028 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:59.638032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:59.638036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:59.638066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:59.640655 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.641319 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.641323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.641325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.641439 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.641787 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.641790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.641792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.642025 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:59.642128 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.642855 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.643097 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:13:59.644336 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.625982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:38117)
0928 05:13:59.626811 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.626860 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.628455 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.629101 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.629111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.629650 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:13:59.630323 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.630363 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.631766 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.631772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.633425 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.633443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.634349 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.634355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.634600 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.635199 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:13:59.635220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:13:59.637855 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:13:59.640785 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:59.640791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:59.640804 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:59.641217 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:59.641529 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:13:59.641532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:13:59.641534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:13:59.641699 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:13:59.642147 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:13:59.642153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:13:59.642450 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.642755 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.643114 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":301,"threads_started":1}
I20260928 05:13:59.645869 14742 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35525
I20260928 05:13:59.646165 14742 heartbeater.cc:464] Registering TS with master...
I20260928 05:13:59.646924 14742 heartbeater.cc:511] Master 127.0.97.62:35525 requested a full tablet report, sending...
I20260928 05:13:59.648440 14285 ts_manager.cc:194] Registered new tserver with Master: ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3:36475)
I20260928 05:13:59.649298 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38117
I20260928 05:13:59.658141 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:13:59.674598 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.662761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.97.62:35525)
0928 05:13:59.662944 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.662948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.663347 (+ 399us) server_negotiation.cc:407] Connection header received
0928 05:13:59.663483 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.663486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.663545 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.663628 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:59.664946 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.665726 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.667050 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.667458 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.667552 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.669911 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:59.669916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:59.669919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:59.669945 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:59.672362 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.672986 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.672990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.672992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.673102 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.673421 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.673424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.673426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.673731 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:59.673919 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.674169 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.674328 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:13:59.678330 14285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:13:59.685388 14285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:13:59.717941 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.697115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36475 (local address 127.0.0.1:41470)
0928 05:13:59.708882 (+ 11767us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.708932 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.709123 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.710611 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.710618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.710665 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:59.711187 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.711202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.713198 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.713205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.715247 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.715258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.716952 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.716958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.717143 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.717195 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.717320 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.717515 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11522,"mutex_wait_us":1,"thread_start_us":13638,"threads_started":1}
I20260928 05:13:59.717960 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.697488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.0.97.3:36475)
0928 05:13:59.709848 (+ 12360us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.709858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.709898 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:13:59.710005 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.710012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.710324 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.711396 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:59.711684 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.713050 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.715403 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.716774 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.716923 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.717165 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.717402 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.717567 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12184,"thread_start_us":12021,"threads_started":1}
I20260928 05:13:59.718973 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.697468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43551 (local address 127.0.0.1:33192)
0928 05:13:59.711187 (+ 13719us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.711205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.711330 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.712021 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.712026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.712054 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:59.712511 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.712526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.713997 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.714005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.716096 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.716119 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.718204 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.718209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.718359 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.718390 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.718488 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.718632 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13533,"thread_start_us":11041,"threads_started":1}
I20260928 05:13:59.719137 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.697993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.97.2:43551)
0928 05:13:59.710162 (+ 12169us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.710170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.711444 (+ 1274us) server_negotiation.cc:407] Connection header received
0928 05:13:59.711549 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.711559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.711814 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.712015 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:59.712736 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.713864 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.716286 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.717725 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.717848 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.718054 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.718580 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.718759 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12022,"thread_start_us":11880,"threads_started":1}
I20260928 05:13:59.719890 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.698419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.0.97.1:38271)
0928 05:13:59.708312 (+ 9893us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.708326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.711325 (+ 2999us) server_negotiation.cc:407] Connection header received
0928 05:13:59.711636 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.711642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.711825 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.711971 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:59.712894 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.714445 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.716885 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.718599 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.718727 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.718939 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.719468 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.719601 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9665,"thread_start_us":9500,"threads_started":1}
I20260928 05:13:59.720563 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.697784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38271 (local address 127.0.0.1:57358)
0928 05:13:59.711194 (+ 13410us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.711213 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.711430 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.712286 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.712291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.712321 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:59.712745 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.712754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.714637 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.714643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.716724 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.716737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.719076 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.719082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.719265 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.719295 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.720138 (+ 843us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.720261 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13270,"thread_start_us":12537,"threads_started":1}
I20260928 05:13:59.722693 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 6951ac0378674db5861cf8c0929a45ca (DEFAULT_TABLE table=test-table [id=9b792b7d3e034f0fbb9c696574f5f3f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:13:59.722698 14677 tablet_service.cc:1467] Processing CreateTablet for tablet 6951ac0378674db5861cf8c0929a45ca (DEFAULT_TABLE table=test-table [id=9b792b7d3e034f0fbb9c696574f5f3f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:13:59.723907 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6951ac0378674db5861cf8c0929a45ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:59.724066 14677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6951ac0378674db5861cf8c0929a45ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:59.724958 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 6951ac0378674db5861cf8c0929a45ca (DEFAULT_TABLE table=test-table [id=9b792b7d3e034f0fbb9c696574f5f3f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:13:59.726207 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6951ac0378674db5861cf8c0929a45ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:13:59.735702 14760 tablet_bootstrap.cc:492] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Bootstrap starting.
I20260928 05:13:59.740159 14760 tablet_bootstrap.cc:654] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:59.741228 14760 log.cc:824] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:59.744062 14760 tablet_bootstrap.cc:492] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: No bootstrap required, opened a new log
I20260928 05:13:59.744379 14760 ts_tablet_manager.cc:1397] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 05:13:59.744395 14761 tablet_bootstrap.cc:492] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Bootstrap starting.
I20260928 05:13:59.746976 14762 tablet_bootstrap.cc:492] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Bootstrap starting.
I20260928 05:13:59.749433 14761 tablet_bootstrap.cc:654] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:59.750373 14762 tablet_bootstrap.cc:654] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Neither blocks nor log segments found. Creating new log.
I20260928 05:13:59.750746 14761 log.cc:824] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:59.751796 14762 log.cc:824] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Log is configured to *not* fsync() on all Append() calls
I20260928 05:13:59.754283 14760 raft_consensus.cc:348] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.754760 14760 raft_consensus.cc:374] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:59.754786 14762 tablet_bootstrap.cc:492] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: No bootstrap required, opened a new log
I20260928 05:13:59.754865 14760 raft_consensus.cc:729] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75c30d5de58454c839d8cb85630294e, State: Initialized, Role: FOLLOWER
I20260928 05:13:59.755064 14762 ts_tablet_manager.cc:1397] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 05:13:59.755232 14761 tablet_bootstrap.cc:492] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: No bootstrap required, opened a new log
I20260928 05:13:59.755496 14761 ts_tablet_manager.cc:1397] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 05:13:59.755577 14760 consensus_queue.cc:260] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.756719 14609 heartbeater.cc:503] Master 127.0.97.62:35525 was elected leader, sending a full tablet report...
I20260928 05:13:59.761451 14760 ts_tablet_manager.cc:1428] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260928 05:13:59.764483 14762 raft_consensus.cc:348] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.764927 14762 raft_consensus.cc:374] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:59.765044 14762 raft_consensus.cc:729] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bfc9ba215d4419a08fc3757806dc39, State: Initialized, Role: FOLLOWER
I20260928 05:13:59.765615 14762 consensus_queue.cc:260] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.766076 14761 raft_consensus.cc:348] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.766424 14761 raft_consensus.cc:374] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:13:59.766563 14761 raft_consensus.cc:729] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba13fcdd7cc54c58a6bc3e5217bcbb8e, State: Initialized, Role: FOLLOWER
I20260928 05:13:59.767154 14761 consensus_queue.cc:260] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.767416 14762 ts_tablet_manager.cc:1428] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 05:13:59.768440 14742 heartbeater.cc:503] Master 127.0.97.62:35525 was elected leader, sending a full tablet report...
I20260928 05:13:59.769093 14761 ts_tablet_manager.cc:1428] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 05:13:59.801220 14768 raft_consensus.cc:482] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:13:59.801522 14768 raft_consensus.cc:504] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.803582 14768 leader_election.cc:288] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e75c30d5de58454c839d8cb85630294e (127.0.97.2:43551), 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271)
I20260928 05:13:59.809899 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.804121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50627 (local address 127.0.97.1:38271)
0928 05:13:59.804312 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.804319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.804340 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:13:59.804419 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.804426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.804515 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.804611 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:59.805097 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.806091 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.807945 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.809116 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.809218 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.809290 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.809630 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.809722 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:13:59.810019 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.803828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38271 (local address 127.0.97.3:50627)
0928 05:13:59.804136 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.804157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.804287 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.804666 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.804671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.804705 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:59.804919 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.804927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.806228 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.806234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.807795 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.807806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.809271 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.809276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.809461 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.809497 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.809594 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.809736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1}
I20260928 05:13:59.810904 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.804055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43291 (local address 127.0.97.2:43551)
0928 05:13:59.804315 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.804325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.805015 (+ 690us) server_negotiation.cc:407] Connection header received
0928 05:13:59.805155 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.805159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.805223 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.805288 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:13:59.805881 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.806870 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.808924 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.810011 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.810144 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.810244 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.810598 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.810738 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 05:13:59.811091 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.804004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43551 (local address 127.0.97.3:43291)
0928 05:13:59.804901 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 05:13:59.804915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:13:59.805036 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:13:59.805332 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:13:59.805337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:13:59.805364 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:13:59.805719 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.805731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.807016 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.807020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.808759 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:13:59.808775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.810279 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.810286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:13:59.810432 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.810453 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 05:13:59.810572 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 05:13:59.810706 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":1,"thread_start_us":507,"threads_started":1}
I20260928 05:13:59.811152 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ac0378674db5861cf8c0929a45ca" candidate_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bfc9ba215d4419a08fc3757806dc39" is_pre_election: true
I20260928 05:13:59.811722 14430 raft_consensus.cc:2454] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba13fcdd7cc54c58a6bc3e5217bcbb8e in term 0.
I20260928 05:13:59.812161 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ac0378674db5861cf8c0929a45ca" candidate_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75c30d5de58454c839d8cb85630294e" is_pre_election: true
I20260928 05:13:59.812350 14633 leader_election.cc:302] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56bfc9ba215d4419a08fc3757806dc39, ba13fcdd7cc54c58a6bc3e5217bcbb8e; no voters:
I20260928 05:13:59.812652 14564 raft_consensus.cc:2454] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba13fcdd7cc54c58a6bc3e5217bcbb8e in term 0.
I20260928 05:13:59.812788 14768 raft_consensus.cc:2781] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:13:59.812971 14768 raft_consensus.cc:482] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:13:59.813113 14768 raft_consensus.cc:3037] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:59.815408 14768 raft_consensus.cc:504] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.816037 14768 leader_election.cc:288] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [CANDIDATE]: Term 1 election: Requested vote from peers e75c30d5de58454c839d8cb85630294e (127.0.97.2:43551), 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271)
I20260928 05:13:59.816397 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ac0378674db5861cf8c0929a45ca" candidate_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75c30d5de58454c839d8cb85630294e"
I20260928 05:13:59.816468 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6951ac0378674db5861cf8c0929a45ca" candidate_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bfc9ba215d4419a08fc3757806dc39"
I20260928 05:13:59.816618 14564 raft_consensus.cc:3037] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:59.816648 14430 raft_consensus.cc:3037] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:13:59.819861 14564 raft_consensus.cc:2454] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba13fcdd7cc54c58a6bc3e5217bcbb8e in term 1.
I20260928 05:13:59.819857 14430 raft_consensus.cc:2454] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba13fcdd7cc54c58a6bc3e5217bcbb8e in term 1.
I20260928 05:13:59.820318 14632 leader_election.cc:302] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba13fcdd7cc54c58a6bc3e5217bcbb8e, e75c30d5de58454c839d8cb85630294e; no voters:
I20260928 05:13:59.820655 14768 raft_consensus.cc:2781] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:13:59.821127 14768 raft_consensus.cc:686] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 LEADER]: Becoming Leader. State: Replica: ba13fcdd7cc54c58a6bc3e5217bcbb8e, State: Running, Role: LEADER
I20260928 05:13:59.821638 14768 consensus_queue.cc:237] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:13:59.826382 14286 catalog_manager.cc:5495] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e reported cstate change: term changed from 0 to 1, leader changed from <none> to ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } health_report { overall_health: UNKNOWN } } }
W20260928 05:13:59.850276 14476 tablet.cc:2369] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:59.863561 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.858009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.97.3:36475)
0928 05:13:59.858250 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.858256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.858275 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:13:59.858465 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.858469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.858592 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.858738 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:13:59.859106 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.859975 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.861432 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.861830 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.861962 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.862457 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:13:59.863003 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:13:59.863169 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.863219 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.863306 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 05:13:59.866199 14610 tablet.cc:2369] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:59.871301 14564 raft_consensus.cc:1261] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Refusing update from remote peer ba13fcdd7cc54c58a6bc3e5217bcbb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:13:59.871305 14430 raft_consensus.cc:1261] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Refusing update from remote peer ba13fcdd7cc54c58a6bc3e5217bcbb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:13:59.872232 14770 consensus_queue.cc:1035] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:13:59.872565 14768 consensus_queue.cc:1035] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:13:59.874495 14743 tablet.cc:2369] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:13:59.880980 14773 mvcc.cc:205] Tried to move back new op lower bound from 7334184713703448576 to 7334184713513533440. Current Snapshot: MvccSnapshot[applied={T|T < 7334184713703448576}]
I20260928 05:13:59.886894 14774 mvcc.cc:205] Tried to move back new op lower bound from 7334184713703448576 to 7334184713513533440. Current Snapshot: MvccSnapshot[applied={T|T < 7334184713703448576}]
I20260928 05:13:59.895848 14286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:13:59.896111 14286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:13:59.899227 14286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Sending DeleteTablet for 3 replicas of tablet 6951ac0378674db5861cf8c0929a45ca
I20260928 05:13:59.900277 14677 tablet_service.cc:1514] Processing DeleteTablet for tablet 6951ac0378674db5861cf8c0929a45ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:13:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41470
I20260928 05:13:59.900311 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet 6951ac0378674db5861cf8c0929a45ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:13:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57358
I20260928 05:13:59.900311 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet 6951ac0378674db5861cf8c0929a45ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:13:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33192
I20260928 05:13:59.903514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14347
I20260928 05:13:59.914353 14784 tablet_replica.cc:330] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: stopping tablet replica
I20260928 05:13:59.915071 14783 tablet_replica.cc:330] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: stopping tablet replica
I20260928 05:13:59.915548 14783 raft_consensus.cc:2229] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:13:59.916119 14783 raft_consensus.cc:2258] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:13:59.916875 14784 raft_consensus.cc:2229] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:13:59.917387 14784 raft_consensus.cc:2258] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:13:59.920018 14783 ts_tablet_manager.cc:1905] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:13:59.920383 14782 tablet_replica.cc:330] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: stopping tablet replica
I20260928 05:13:59.920796 14782 raft_consensus.cc:2229] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:13:59.921474 14782 raft_consensus.cc:2258] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:13:59.923285 14782 ts_tablet_manager.cc:1905] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:13:59.923309 14784 ts_tablet_manager.cc:1905] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:13:59 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:13:59.930757 14784 ts_tablet_manager.cc:1918] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:13:59.930932 14784 log.cc:1196] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/6951ac0378674db5861cf8c0929a45ca
I20260928 05:13:59.930928 14782 ts_tablet_manager.cc:1918] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:13:59.931106 14782 log.cc:1196] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/6951ac0378674db5861cf8c0929a45ca
I20260928 05:13:59.931483 14784 ts_tablet_manager.cc:1939] T 6951ac0378674db5861cf8c0929a45ca P 56bfc9ba215d4419a08fc3757806dc39: Deleting consensus metadata
I20260928 05:13:59.931675 14782 ts_tablet_manager.cc:1939] T 6951ac0378674db5861cf8c0929a45ca P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Deleting consensus metadata
I20260928 05:13:59.933508 14270 catalog_manager.cc:4841] TS ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3:36475): tablet 6951ac0378674db5861cf8c0929a45ca (table test-table [id=9b792b7d3e034f0fbb9c696574f5f3f0]) successfully deleted
I20260928 05:13:59.934689 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:13:59.904356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.97.1:38271)
0928 05:13:59.904554 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:13:59.904563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:13:59.911449 (+ 6886us) server_negotiation.cc:407] Connection header received
0928 05:13:59.912572 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:13:59.912582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:13:59.912719 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:13:59.912820 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:13:59.915813 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.916910 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.919651 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:13:59.920239 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:13:59.920340 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:13:59.926560 (+ 6220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:13:59.926576 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:13:59.926592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:13:59.926633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:13:59.930630 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.931942 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.931952 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.931955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.932112 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:13:59.932562 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:13:59.932567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:13:59.932570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:13:59.933049 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:13:59.933337 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 05:13:59.934056 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 05:13:59.934287 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 05:13:59.937567 14470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:13:59.939138 14783 ts_tablet_manager.cc:1918] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:13:59.939352 14783 log.cc:1196] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/6951ac0378674db5861cf8c0929a45ca
I20260928 05:13:59.940275 14783 ts_tablet_manager.cc:1939] T 6951ac0378674db5861cf8c0929a45ca P e75c30d5de58454c839d8cb85630294e: Deleting consensus metadata
I20260928 05:13:59.942498 14272 catalog_manager.cc:4841] TS e75c30d5de58454c839d8cb85630294e (127.0.97.2:43551): tablet 6951ac0378674db5861cf8c0929a45ca (table test-table [id=9b792b7d3e034f0fbb9c696574f5f3f0]) successfully deleted
I20260928 05:14:00.059931 14273 catalog_manager.cc:4841] TS 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271): tablet 6951ac0378674db5861cf8c0929a45ca (table test-table [id=9b792b7d3e034f0fbb9c696574f5f3f0]) successfully deleted
I20260928 05:14:00.060276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14347
I20260928 05:14:00.078716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14481
Sep 28 05:14:00 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:14:00.095033 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:00.079559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.0.97.2:43551)
0928 05:14:00.079754 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:00.079762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:00.079788 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:14:00.080008 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:00.080014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:00.080105 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:00.080215 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:00.081482 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:00.082205 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:00.083786 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:00.084170 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:00.084312 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:00.088872 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:00.088886 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:00.088901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:00.088933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:00.092200 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:00.093080 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:00.093087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:00.093089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:00.093210 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:00.093605 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:00.093608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:00.093610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:00.093959 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:00.094163 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:00.094481 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 05:14:00.094707 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 05:14:00.095651 14604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:00.188405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14481
I20260928 05:14:00.211283 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14614
Sep 28 05:14:00 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:14:00.228269 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:00.212106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.97.3:36475)
0928 05:14:00.212309 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:00.212315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:00.212334 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:14:00.212493 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:00.212499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:00.212572 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:00.212684 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:00.213955 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:00.214900 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:00.216217 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:00.216683 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:00.216842 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:00.221967 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:00.221981 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:00.221997 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:00.222043 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:00.225143 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:00.226007 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:00.226013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:00.226016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:00.226171 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:00.226759 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:00.226762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:00.226764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:00.227125 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:00.227371 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:00.227733 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 05:14:00.227900 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:14:00.229029 14737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:00.352538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14614
I20260928 05:14:00.368630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14254
I20260928 05:14:00.369830 14316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:00.479383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14254
I20260928 05:14:00.500317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35525
--webserver_interface=127.0.97.62
--webserver_port=34587
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:00.823119 14790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:00.823416 14790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:00.823585 14790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:00.830801 14790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:14:00.830929 14790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:00.831037 14790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:00.831135 14790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:14:00.831221 14790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:14:00.841082 14790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:00.842381 14790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:00.843714 14790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:00.866266 14797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:00.866343 14799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:00.866365 14796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:00.872030 14790 server_base.cc:1034] running on GCE node
I20260928 05:14:00.872968 14790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:00.874811 14790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:00.876065 14790 hybrid_clock.cc:648] HybridClock initialized: now 1790572440875993 us; error 55 us; skew 500 ppm
I20260928 05:14:00.876516 14790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:00 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:00.880471 14790 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:14:00.882293 14790 webserver.cc:460] Webserver started at http://127.0.97.62:34587/ using document root <none> and password file <none>
I20260928 05:14:00.883198 14790 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:00.883375 14790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:00.890213 14790 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:14:00.893363 14806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:00.894867 14790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:14:00.895066 14790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "ec0aaf2b787f4b59b05d2f2078542ba1"
format_stamp: "Formatted at 2026-09-28 05:13:57 on dist-test-slave-6718"
server_key: "51bfaf8659d21d40d0dcf5814235e4c8"
server_key_iv: "cc3ed5e701d457bb4e0ab2a5108c6e3c"
server_key_version: "encryptionkey@0"
I20260928 05:14:00.896126 14790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:00.917089 14790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:00.921546 14790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:00.921867 14790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:00.948524 14790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35525
I20260928 05:14:00.948482 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35525 every 8 connection(s)
I20260928 05:14:00.950286 14790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:14:00.951756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14790
I20260928 05:14:00.952536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38271
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35525
--webserver_port=42361
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:14:00.957796 14858 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:14:00.962014 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap starting.
I20260928 05:14:00.981664 14858 log.cc:824] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:00.993383 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:14:00.993927 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap complete.
I20260928 05:14:01.003640 14858 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:01.004521 14858 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0aaf2b787f4b59b05d2f2078542ba1, State: Initialized, Role: FOLLOWER
I20260928 05:14:01.005172 14858 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:01.005426 14858 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:14:01.005559 14858 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:14:01.005733 14858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:14:01.009512 14858 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:01.010035 14858 leader_election.cc:302] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0aaf2b787f4b59b05d2f2078542ba1; no voters:
I20260928 05:14:01.010712 14858 leader_election.cc:288] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:14:01.010874 14862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:14:01.011986 14862 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 LEADER]: Becoming Leader. State: Replica: ec0aaf2b787f4b59b05d2f2078542ba1, State: Running, Role: LEADER
I20260928 05:14:01.012564 14862 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:01.013038 14858 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:14:01.014966 14864 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0aaf2b787f4b59b05d2f2078542ba1. Latest consensus state: current_term: 2 leader_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } } }
I20260928 05:14:01.015435 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } } }
I20260928 05:14:01.015575 14864 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:01.015710 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:01.028566 14868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:14:01.036763 14868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:14:01.038481 14868 catalog_manager.cc:1259] Loaded cluster ID: db8e19ca0e0c4121bb5e384be1416289
I20260928 05:14:01.038628 14868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:14:01.041796 14868 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 05:14:01.059226 14868 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 05:14:01.068718 14868 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:14:01.070390 14868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Loaded TSK: 0
I20260928 05:14:01.071949 14868 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:14:01.297447 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:01.297745 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:01.297920 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:01.305269 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:01.305439 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:01.305613 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:14:01.315621 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42361
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:01.316943 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:01.318259 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:01.331485 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:01.331835 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:01.331934 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:01.333454 14860 server_base.cc:1034] running on GCE node
I20260928 05:14:01.334121 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:01.335248 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:01.336473 14860 hybrid_clock.cc:648] HybridClock initialized: now 1790572441336404 us; error 57 us; skew 500 ppm
I20260928 05:14:01.336845 14860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:01 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:01.340349 14860 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:14:01.341974 14860 webserver.cc:460] Webserver started at http://127.0.97.1:42361/ using document root <none> and password file <none>
I20260928 05:14:01.342796 14860 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:01.342955 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:01.348476 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:14:01.351338 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:01.352730 14860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:14:01.352907 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "56bfc9ba215d4419a08fc3757806dc39"
format_stamp: "Formatted at 2026-09-28 05:13:58 on dist-test-slave-6718"
server_key: "8fb33f98c78c8749be4b606a7f138987"
server_key_iv: "286fa6f4b796ea9e9909ff3a8e5774f0"
server_key_version: "encryptionkey@0"
I20260928 05:14:01.353880 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:01.377048 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:01.381319 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:01.381599 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:01.382644 14860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:01.384075 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:01.384204 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:01.384332 14860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:01.384418 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:01.417968 14860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38271
I20260928 05:14:01.418025 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38271 every 8 connection(s)
I20260928 05:14:01.419447 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:14:01.427115 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 14860
I20260928 05:14:01.427732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43551
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35525
--webserver_port=40327
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:14:01 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:01.445572 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:01.422524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:52551)
0928 05:14:01.423417 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:01.423467 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:01.425140 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:01.426488 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:01.426496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:01.427115 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:01.427758 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:01.427777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.429610 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.429619 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:01.431829 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:01.431848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.432934 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.432943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:01.433245 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:01.434012 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:01.434050 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:01.436320 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:01.441352 (+ 5032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:01.441359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:01.441370 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:01.441758 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:01.442293 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:01.442296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:01.442298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:01.442458 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:01.443499 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:01.443505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:01.443832 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 05:14:01.444076 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 05:14:01.444500 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":375,"threads_started":1}
I20260928 05:14:01.446244 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:01.422279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52551 (local address 127.0.97.62:35525)
0928 05:14:01.422853 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:01.422864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:01.423591 (+ 727us) server_negotiation.cc:407] Connection header received
0928 05:14:01.425643 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:01.425655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:01.425963 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:01.426244 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:01.428097 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.429438 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.431989 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.432686 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.432837 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:01.436611 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:01.436624 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:01.436655 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:01.436715 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:01.441191 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:01.441930 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:01.441942 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:01.441954 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:01.442178 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:01.442606 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:01.442615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:01.442622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:01.443317 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:01.443548 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:01.444373 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 05:14:01.444889 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
I20260928 05:14:01.449523 15008 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35525
I20260928 05:14:01.449903 15008 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:01.450973 15008 heartbeater.cc:511] Master 127.0.97.62:35525 requested a full tablet report, sending...
I20260928 05:14:01.453779 14823 ts_manager.cc:194] Registered new tserver with Master: 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271)
I20260928 05:14:01.456362 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52551
W20260928 05:14:01.759752 15012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:01.760105 15012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:01.760210 15012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:01.767586 15012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:01.767709 15012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:01.767819 15012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:14:01.778025 15012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40327
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:01.779330 15012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:01.780548 15012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:01.794337 15018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:01.794269 15020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:01.794385 15017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:01.800135 15012 server_base.cc:1034] running on GCE node
I20260928 05:14:01.800772 15012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:01.801797 15012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:01.802990 15012 hybrid_clock.cc:648] HybridClock initialized: now 1790572441802947 us; error 33 us; skew 500 ppm
I20260928 05:14:01.803269 15012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:01 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:01.809712 15012 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:14:01.811223 15012 webserver.cc:460] Webserver started at http://127.0.97.2:40327/ using document root <none> and password file <none>
I20260928 05:14:01.811884 15012 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:01.811990 15012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:01.817412 15012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:14:01.820103 15027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:01.821451 15012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:14:01.821633 15012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "e75c30d5de58454c839d8cb85630294e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "eeb60f01854f62affd4bc8e50b516d31"
server_key_iv: "63bd7e5a6443b6016c3415c388d5ab40"
server_key_version: "encryptionkey@0"
I20260928 05:14:01.822628 15012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:01.840760 15012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:01.844930 15012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:01.845217 15012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:01.846215 15012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:01.847539 15012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:01.847628 15012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:01.847779 15012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:01.847870 15012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:01.880405 15012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43551
I20260928 05:14:01.880492 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43551 every 8 connection(s)
I20260928 05:14:01.881878 15012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:14:01.884217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15012
I20260928 05:14:01.884925 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36475
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35525
--webserver_port=40683
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:14:01 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:01.904150 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:01.884486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38179 (local address 127.0.97.62:35525)
0928 05:14:01.884711 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:01.884719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:01.886005 (+ 1286us) server_negotiation.cc:407] Connection header received
0928 05:14:01.887929 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:01.887938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:01.888036 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:01.888168 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:01.889610 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.890766 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.892288 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.893050 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.893193 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:01.896597 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:01.896607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:01.896614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:01.896664 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:01.900375 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:01.901276 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:01.901284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:01.901290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:01.901456 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:01.901978 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:01.901986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:01.901992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:01.902433 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:01.902633 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:01.903394 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 05:14:01.903680 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:14:01.904820 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:01.884948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:38179)
0928 05:14:01.885825 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:01.885868 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:01.887652 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:01.888380 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:01.888388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:01.888874 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:01.889432 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:01.889450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.890922 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.890926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:01.892141 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:01.892152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:01.893246 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:01.893250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:01.893460 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:01.894132 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:01.894153 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:01.896371 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:01.900560 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:01.900568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:01.900583 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:01.901130 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:01.901596 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:01.901600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:01.901603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:01.901814 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:01.902637 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:01.902644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:01.902976 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 05:14:01.903236 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 05:14:01.903668 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":382,"threads_started":1}
I20260928 05:14:01.906419 15140 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35525
I20260928 05:14:01.906813 15140 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:01.907810 15140 heartbeater.cc:511] Master 127.0.97.62:35525 requested a full tablet report, sending...
I20260928 05:14:01.909477 14823 ts_manager.cc:194] Registered new tserver with Master: e75c30d5de58454c839d8cb85630294e (127.0.97.2:43551)
I20260928 05:14:01.910301 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38179
W20260928 05:14:02.212862 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:02.213176 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:02.213356 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:02.220508 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:02.220634 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:02.220825 15143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:14:02.230715 15143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40683
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:02.231957 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:02.233247 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:02.246351 15151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:02.246402 15148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:02.246322 15149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:02.247193 15143 server_base.cc:1034] running on GCE node
I20260928 05:14:02.247805 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:02.248757 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:02.250013 15143 hybrid_clock.cc:648] HybridClock initialized: now 1790572442249959 us; error 45 us; skew 500 ppm
I20260928 05:14:02.250365 15143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:02 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:02.253943 15143 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:14:02.255465 15143 webserver.cc:460] Webserver started at http://127.0.97.3:40683/ using document root <none> and password file <none>
I20260928 05:14:02.256188 15143 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:02.256354 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:02.261943 15143 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 05:14:02.264693 15158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:02.266081 15143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:14:02.266283 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "012d88fd608b83b5bbfdb3d9f392986d"
server_key_iv: "7f33ee2ed15e2a8ec0455ba158e5ce22"
server_key_version: "encryptionkey@0"
I20260928 05:14:02.267351 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:02.284128 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:02.287993 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:02.288271 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:02.289222 15143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:02.290491 15143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:02.290637 15143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:02.290777 15143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:02.290866 15143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:02.323915 15143 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36475
I20260928 05:14:02.324034 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36475 every 8 connection(s)
I20260928 05:14:02.325421 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:14:02.329663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15143
Sep 28 05:14:02 dist-test-slave-6718 krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:02.345788 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.330682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.97.62:35525)
0928 05:14:02.332567 (+ 1885us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.332573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.332588 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:14:02.332632 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.332636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.332699 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.332811 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:02.334369 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.335462 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.336829 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.337308 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.337437 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.340016 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:02.340021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:02.340024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:02.340052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:02.342901 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:02.343672 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:02.343676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:02.343678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:02.343787 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:02.344177 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:02.344182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:02.344185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:02.344522 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:02.344777 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.345095 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.345320 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":1744,"thread_start_us":1651,"threads_started":1}
I20260928 05:14:02.347222 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.328312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35635 (local address 127.0.97.62:35525)
0928 05:14:02.328516 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.328522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.329919 (+ 1397us) server_negotiation.cc:407] Connection header received
0928 05:14:02.332265 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.332274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.332384 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.332494 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:02.334065 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.335090 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.336723 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.337230 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.337324 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.340755 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:02.340763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:02.340768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:02.340827 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:02.343860 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:02.344672 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:02.344677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:02.344680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:02.344830 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:02.345327 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:02.345332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:02.345335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:02.345652 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:02.345757 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.346681 (+ 924us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.346911 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":66}
I20260928 05:14:02.348169 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.328899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:35635)
0928 05:14:02.329732 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:02.329792 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:02.331989 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:02.332785 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:02.332798 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:02.333337 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:02.333880 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.333897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.335253 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.335259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.336581 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.336594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.337454 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.337462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.337715 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.338392 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:02.338412 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:02.340557 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:02.343992 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:02.343998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:02.344008 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:02.344563 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:02.344939 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:02.344942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:02.344944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:02.345189 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:02.345787 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:02.345795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:02.346176 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 05:14:02.346558 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 05:14:02.346939 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":316,"threads_started":1}
I20260928 05:14:02.349437 15271 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35525
I20260928 05:14:02.349726 15271 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:02.350412 15271 heartbeater.cc:511] Master 127.0.97.62:35525 requested a full tablet report, sending...
I20260928 05:14:02.351771 14823 ts_manager.cc:194] Registered new tserver with Master: ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3:36475)
I20260928 05:14:02.352521 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35635
I20260928 05:14:02.360641 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:14:02.367468 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.361504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.0.97.62:35525)
0928 05:14:02.361663 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.361668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.362150 (+ 482us) server_negotiation.cc:407] Connection header received
0928 05:14:02.362320 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.362324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.362452 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.362665 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:14:02.363144 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.363849 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.365315 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.365717 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.365862 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.366386 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:14:02.366906 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:14:02.367075 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.367132 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.367210 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:14:02.368291 14823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:14:02.375382 14823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:14:02.399835 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.386782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.0.97.2:43551)
0928 05:14:02.392204 (+ 5422us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.392211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.392239 (+ 28us) server_negotiation.cc:407] Connection header received
0928 05:14:02.392321 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.392327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.392489 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.392718 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:02.393355 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.394373 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.397531 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.398662 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.398794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.399031 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.399302 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.399469 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5266,"thread_start_us":5154,"threads_started":1}
I20260928 05:14:02.399914 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.386164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43551 (local address 127.0.0.1:33204)
0928 05:14:02.387664 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:02.387718 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:02.387904 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:02.392733 (+ 4829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:02.392745 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:02.392773 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:02.393147 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.393159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.394527 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.394552 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.397340 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.397356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.398928 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.398934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.399088 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.399138 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 05:14:02.399234 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:14:02.399392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"mutex_wait_us":134,"thread_start_us":1078,"threads_started":1}
I20260928 05:14:02.400650 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.387212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.0.97.3:36475)
0928 05:14:02.388563 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.388575 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.389641 (+ 1066us) server_negotiation.cc:407] Connection header received
0928 05:14:02.389762 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.389773 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.390084 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.390314 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:02.392622 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.394284 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.397027 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.398215 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.398350 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.398612 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.400156 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.400325 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":983,"threads_started":1}
I20260928 05:14:02.400677 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.386828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38271 (local address 127.0.0.1:57372)
0928 05:14:02.391973 (+ 5145us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:02.391991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:02.392099 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:02.392952 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:02.392958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:02.392992 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:02.393371 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.393382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.395291 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.395296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.397465 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.397479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.399386 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.399391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.399613 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.399657 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 05:14:02.399739 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 05:14:02.399870 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4980,"mutex_wait_us":623,"thread_start_us":1551,"threads_started":1}
I20260928 05:14:02.399569 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.386815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36475 (local address 127.0.0.1:41504)
0928 05:14:02.389330 (+ 2515us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:02.389352 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:02.389501 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:02.391435 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:02.391447 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:02.391489 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:02.392455 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.392464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.394436 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.394444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.396865 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.396877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.398424 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.398431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.398663 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.398723 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 05:14:02.398867 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 05:14:02.399071 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343,"mutex_wait_us":130,"thread_start_us":4781,"threads_started":1}
I20260928 05:14:02.401289 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.387295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.0.97.1:38271)
0928 05:14:02.391111 (+ 3816us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.391123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.392137 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 05:14:02.392241 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.392249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.392491 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.392678 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:02.393542 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.395095 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.397608 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.399061 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.399193 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.400131 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.400200 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.400964 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3597,"thread_start_us":3459,"threads_started":1}
I20260928 05:14:02.404459 14943 tablet_service.cc:1467] Processing CreateTablet for tablet ffb3a27d19be4cbfac945d2a11c94b38 (DEFAULT_TABLE table=test-table [id=f61faf96ccc2479ea2ab6fd5c0a0c968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:02.404637 15075 tablet_service.cc:1467] Processing CreateTablet for tablet ffb3a27d19be4cbfac945d2a11c94b38 (DEFAULT_TABLE table=test-table [id=f61faf96ccc2479ea2ab6fd5c0a0c968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:02.404636 15206 tablet_service.cc:1467] Processing CreateTablet for tablet ffb3a27d19be4cbfac945d2a11c94b38 (DEFAULT_TABLE table=test-table [id=f61faf96ccc2479ea2ab6fd5c0a0c968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:02.405925 14943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb3a27d19be4cbfac945d2a11c94b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:02.406031 15206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb3a27d19be4cbfac945d2a11c94b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:02.406189 15075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb3a27d19be4cbfac945d2a11c94b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:02.424093 15287 tablet_bootstrap.cc:492] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Bootstrap starting.
I20260928 05:14:02.427330 15289 tablet_bootstrap.cc:492] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Bootstrap starting.
I20260928 05:14:02.428787 15288 tablet_bootstrap.cc:492] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Bootstrap starting.
I20260928 05:14:02.430091 15287 tablet_bootstrap.cc:654] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:02.431497 15287 log.cc:824] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:02.432116 15289 tablet_bootstrap.cc:654] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:02.432188 15288 tablet_bootstrap.cc:654] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:02.433414 15288 log.cc:824] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:02.433414 15289 log.cc:824] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:02.440646 15287 tablet_bootstrap.cc:492] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: No bootstrap required, opened a new log
I20260928 05:14:02.440999 15287 ts_tablet_manager.cc:1397] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260928 05:14:02.441233 15289 tablet_bootstrap.cc:492] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: No bootstrap required, opened a new log
I20260928 05:14:02.441491 15289 ts_tablet_manager.cc:1397] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 05:14:02.442414 15288 tablet_bootstrap.cc:492] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: No bootstrap required, opened a new log
I20260928 05:14:02.442730 15288 ts_tablet_manager.cc:1397] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260928 05:14:02.449020 15289 raft_consensus.cc:348] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.449404 15289 raft_consensus.cc:374] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:02.449546 15289 raft_consensus.cc:729] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba13fcdd7cc54c58a6bc3e5217bcbb8e, State: Initialized, Role: FOLLOWER
I20260928 05:14:02.450174 15289 consensus_queue.cc:260] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.451368 15271 heartbeater.cc:503] Master 127.0.97.62:35525 was elected leader, sending a full tablet report...
I20260928 05:14:02.452047 15289 ts_tablet_manager.cc:1428] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260928 05:14:02.451817 15287 raft_consensus.cc:348] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.452279 15287 raft_consensus.cc:374] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:02.452370 15287 raft_consensus.cc:729] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bfc9ba215d4419a08fc3757806dc39, State: Initialized, Role: FOLLOWER
I20260928 05:14:02.453004 15287 consensus_queue.cc:260] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.453176 15288 raft_consensus.cc:348] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.453660 15288 raft_consensus.cc:374] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:02.453806 15288 raft_consensus.cc:729] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75c30d5de58454c839d8cb85630294e, State: Initialized, Role: FOLLOWER
I20260928 05:14:02.454196 15008 heartbeater.cc:503] Master 127.0.97.62:35525 was elected leader, sending a full tablet report...
I20260928 05:14:02.454414 15288 consensus_queue.cc:260] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.456197 15287 ts_tablet_manager.cc:1428] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260928 05:14:02.457916 15140 heartbeater.cc:503] Master 127.0.97.62:35525 was elected leader, sending a full tablet report...
I20260928 05:14:02.458384 15288 ts_tablet_manager.cc:1428] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260928 05:14:02.577636 15272 tablet.cc:2369] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:14:02.633780 15141 tablet.cc:2369] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:14:02.671785 15009 tablet.cc:2369] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:14:02.742926 15295 raft_consensus.cc:482] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:14:02.743212 15295 raft_consensus.cc:504] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.745276 15295 leader_election.cc:288] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3:36475), 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271)
I20260928 05:14:02.751786 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.745781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45899 (local address 127.0.97.1:38271)
0928 05:14:02.745995 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.746004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.746472 (+ 468us) server_negotiation.cc:407] Connection header received
0928 05:14:02.746696 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.746700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.746757 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.746852 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:02.747459 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.748167 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.749998 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.750817 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.750963 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.751068 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.751392 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.751520 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 05:14:02.751864 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.745733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32935 (local address 127.0.97.3:36475)
0928 05:14:02.745956 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.745965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.746563 (+ 598us) server_negotiation.cc:407] Connection header received
0928 05:14:02.746792 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.746796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.746853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.746949 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:02.747604 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.748314 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.750052 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.750914 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.751064 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.751172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.751471 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.751596 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:14:02.751967 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.745529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38271 (local address 127.0.97.2:45899)
0928 05:14:02.746361 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:02.746376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:02.746569 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:02.746865 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:02.746870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:02.746892 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:02.747311 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.747320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.748287 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.748294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.749833 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.749845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.751045 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.751051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.751224 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.751248 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:14:02.751356 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:14:02.751496 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":67,"thread_start_us":485,"threads_started":1}
I20260928 05:14:02.752007 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.745553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36475 (local address 127.0.97.2:32935)
0928 05:14:02.746455 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:02.746469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:02.746644 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:02.746976 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:02.746980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:02.747020 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:02.747440 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.747449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.748454 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.748462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.749894 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:02.749905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.751129 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.751135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:02.751315 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.751337 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 05:14:02.751443 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 05:14:02.751572 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":5,"thread_start_us":471,"threads_started":1}
I20260928 05:14:02.753186 15226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb3a27d19be4cbfac945d2a11c94b38" candidate_uuid: "e75c30d5de58454c839d8cb85630294e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" is_pre_election: true
I20260928 05:14:02.753868 15226 raft_consensus.cc:2454] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e75c30d5de58454c839d8cb85630294e in term 0.
I20260928 05:14:02.753881 14963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb3a27d19be4cbfac945d2a11c94b38" candidate_uuid: "e75c30d5de58454c839d8cb85630294e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bfc9ba215d4419a08fc3757806dc39" is_pre_election: true
I20260928 05:14:02.754487 14963 raft_consensus.cc:2454] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e75c30d5de58454c839d8cb85630294e in term 0.
I20260928 05:14:02.754660 15028 leader_election.cc:302] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba13fcdd7cc54c58a6bc3e5217bcbb8e, e75c30d5de58454c839d8cb85630294e; no voters:
I20260928 05:14:02.755088 15295 raft_consensus.cc:2781] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:14:02.755228 15295 raft_consensus.cc:482] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:14:02.755355 15295 raft_consensus.cc:3037] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:02.757899 15295 raft_consensus.cc:504] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.758635 15295 leader_election.cc:288] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [CANDIDATE]: Term 1 election: Requested vote from peers ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3:36475), 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271)
I20260928 05:14:02.758975 15226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb3a27d19be4cbfac945d2a11c94b38" candidate_uuid: "e75c30d5de58454c839d8cb85630294e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e"
I20260928 05:14:02.759115 14963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb3a27d19be4cbfac945d2a11c94b38" candidate_uuid: "e75c30d5de58454c839d8cb85630294e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bfc9ba215d4419a08fc3757806dc39"
I20260928 05:14:02.759191 15226 raft_consensus.cc:3037] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:02.759366 14963 raft_consensus.cc:3037] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:02.762393 15226 raft_consensus.cc:2454] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e75c30d5de58454c839d8cb85630294e in term 1.
I20260928 05:14:02.762477 14963 raft_consensus.cc:2454] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e75c30d5de58454c839d8cb85630294e in term 1.
I20260928 05:14:02.762871 15028 leader_election.cc:302] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba13fcdd7cc54c58a6bc3e5217bcbb8e, e75c30d5de58454c839d8cb85630294e; no voters:
I20260928 05:14:02.763239 15295 raft_consensus.cc:2781] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:14:02.763705 15295 raft_consensus.cc:686] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 1 LEADER]: Becoming Leader. State: Replica: e75c30d5de58454c839d8cb85630294e, State: Running, Role: LEADER
I20260928 05:14:02.764195 15295 consensus_queue.cc:237] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } }
I20260928 05:14:02.768745 14822 catalog_manager.cc:5495] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e reported cstate change: term changed from 0 to 1, leader changed from <none> to e75c30d5de58454c839d8cb85630294e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e75c30d5de58454c839d8cb85630294e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75c30d5de58454c839d8cb85630294e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 } health_report { overall_health: UNKNOWN } } }
I20260928 05:14:02.857293 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.852165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.0.97.2:43551)
0928 05:14:02.852380 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.852385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.852402 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:14:02.852569 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.852573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.852648 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.852770 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:14:02.853061 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.853740 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.855243 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.855634 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.855725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.856144 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:14:02.856655 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:14:02.856765 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.856888 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.856990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 05:14:02.865310 14963 raft_consensus.cc:1261] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Refusing update from remote peer e75c30d5de58454c839d8cb85630294e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:14:02.865327 15226 raft_consensus.cc:1261] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Refusing update from remote peer e75c30d5de58454c839d8cb85630294e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:14:02.866214 15298 consensus_queue.cc:1035] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56bfc9ba215d4419a08fc3757806dc39" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:14:02.866565 15295 consensus_queue.cc:1035] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:14:02.878217 15301 mvcc.cc:205] Tried to move back new op lower bound from 7334184725966979072 to 7334184725566263296. Current Snapshot: MvccSnapshot[applied={T|T < 7334184725966979072}]
I20260928 05:14:02.899227 14822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:14:02.899509 14822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:14:02.903297 14822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Sending DeleteTablet for 3 replicas of tablet ffb3a27d19be4cbfac945d2a11c94b38
I20260928 05:14:02.904438 15206 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb3a27d19be4cbfac945d2a11c94b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41504
I20260928 05:14:02.904446 15075 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb3a27d19be4cbfac945d2a11c94b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33204
I20260928 05:14:02.905205 14943 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb3a27d19be4cbfac945d2a11c94b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57372
I20260928 05:14:02.905376 15311 tablet_replica.cc:330] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: stopping tablet replica
I20260928 05:14:02.905769 15311 raft_consensus.cc:2229] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:14:02.906404 15311 raft_consensus.cc:2258] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:02.906723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14860
I20260928 05:14:02.907114 15312 tablet_replica.cc:330] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: stopping tablet replica
I20260928 05:14:02.907513 15312 raft_consensus.cc:2229] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:14:02.907971 15312 raft_consensus.cc:2258] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:02.908811 15311 ts_tablet_manager.cc:1905] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:02.909896 15312 ts_tablet_manager.cc:1905] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:02.916884 15312 ts_tablet_manager.cc:1918] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:02.917081 15312 log.cc:1196] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/ffb3a27d19be4cbfac945d2a11c94b38
I20260928 05:14:02.917068 15313 tablet_replica.cc:330] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: stopping tablet replica
I20260928 05:14:02.917660 15312 ts_tablet_manager.cc:1939] T ffb3a27d19be4cbfac945d2a11c94b38 P ba13fcdd7cc54c58a6bc3e5217bcbb8e: Deleting consensus metadata
I20260928 05:14:02.917817 15313 raft_consensus.cc:2229] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:14:02.918494 15313 raft_consensus.cc:2258] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:02.919559 14807 catalog_manager.cc:4841] TS ba13fcdd7cc54c58a6bc3e5217bcbb8e (127.0.97.3:36475): tablet ffb3a27d19be4cbfac945d2a11c94b38 (table test-table [id=f61faf96ccc2479ea2ab6fd5c0a0c968]) successfully deleted
I20260928 05:14:02.922643 15311 ts_tablet_manager.cc:1918] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:02.922863 15311 log.cc:1196] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/ffb3a27d19be4cbfac945d2a11c94b38
I20260928 05:14:02.923362 15311 ts_tablet_manager.cc:1939] T ffb3a27d19be4cbfac945d2a11c94b38 P e75c30d5de58454c839d8cb85630294e: Deleting consensus metadata
I20260928 05:14:02.925109 14809 catalog_manager.cc:4841] TS e75c30d5de58454c839d8cb85630294e (127.0.97.2:43551): tablet ffb3a27d19be4cbfac945d2a11c94b38 (table test-table [id=f61faf96ccc2479ea2ab6fd5c0a0c968]) successfully deleted
I20260928 05:14:02.925390 15313 ts_tablet_manager.cc:1905] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:02.932714 15313 ts_tablet_manager.cc:1918] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:02.932894 15313 log.cc:1196] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/ffb3a27d19be4cbfac945d2a11c94b38
I20260928 05:14:02.933526 15313 ts_tablet_manager.cc:1939] T ffb3a27d19be4cbfac945d2a11c94b38 P 56bfc9ba215d4419a08fc3757806dc39: Deleting consensus metadata
I20260928 05:14:02.935523 14810 catalog_manager.cc:4841] TS 56bfc9ba215d4419a08fc3757806dc39 (127.0.97.1:38271): tablet ffb3a27d19be4cbfac945d2a11c94b38 (table test-table [id=f61faf96ccc2479ea2ab6fd5c0a0c968]) successfully deleted
I20260928 05:14:02.938012 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:02.907914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.0.97.1:38271)
0928 05:14:02.908097 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:02.908103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:02.920380 (+ 12277us) server_negotiation.cc:407] Connection header received
0928 05:14:02.920583 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:02.920589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:02.920686 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:02.920818 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:02.922705 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.923792 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.925980 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:02.926822 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:02.926948 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:02.930973 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:02.930999 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:02.931017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:02.931062 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:02.935066 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:02.935863 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:02.935872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:02.935876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:02.936012 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:02.936408 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:02.936413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:02.936417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:02.936780 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:02.937069 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:02.937401 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 05:14:02.937616 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":68}
I20260928 05:14:02.938736 15003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:03.031160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14860
I20260928 05:14:03.049268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15012
I20260928 05:14:03.064018 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:03.050162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.97.2:43551)
0928 05:14:03.050363 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:03.050369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:03.050388 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:14:03.050588 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:03.050594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:03.050675 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:03.050785 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:03.052129 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:03.052919 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:03.054325 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:03.054977 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:03.055120 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:03.057856 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:03.057868 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:03.057883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:03.057926 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:03.061108 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:03.061911 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:03.061917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:03.061919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:03.062062 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:03.062451 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:03.062454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:03.062456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:03.062959 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:03.063193 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:03.063520 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 05:14:03.063672 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:14:03.064769 15135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:03.178818 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15012
I20260928 05:14:03.197953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15143
I20260928 05:14:03.215349 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:03.198729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.0.97.3:36475)
0928 05:14:03.198921 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:03.198926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:03.198944 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:14:03.199077 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:03.199081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:03.199148 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:03.199229 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:03.200607 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:03.201348 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:03.202739 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:03.203117 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:03.203213 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:03.209862 (+ 6649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:03.209878 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:03.209894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:03.209931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:03.212766 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:03.213398 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:03.213405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:03.213407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:03.213587 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:03.213904 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:03.213907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:03.213909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:03.214307 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:03.214519 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:03.214810 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 05:14:03.214982 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:14:03.216033 15266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:03.305946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15143
I20260928 05:14:03.324249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 14790
I20260928 05:14:03.325349 14853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:03.436723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 14790
I20260928 05:14:03.458213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35525
--webserver_interface=127.0.97.62
--webserver_port=34587
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:03.780104 15319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:03.780385 15319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:03.780562 15319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:03.787694 15319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:14:03.787806 15319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:03.787853 15319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:03.787935 15319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:14:03.788017 15319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:14:03.797883 15319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:03.799379 15319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:03.800724 15319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:03.812274 15324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:03.812328 15327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:03.812351 15325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:03.813097 15319 server_base.cc:1034] running on GCE node
I20260928 05:14:03.813907 15319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:03.815310 15319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:03.816545 15319 hybrid_clock.cc:648] HybridClock initialized: now 1790572443816491 us; error 62 us; skew 500 ppm
I20260928 05:14:03.816897 15319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:03 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:03.820092 15319 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:14:03.821471 15319 webserver.cc:460] Webserver started at http://127.0.97.62:34587/ using document root <none> and password file <none>
I20260928 05:14:03.822134 15319 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:03.822278 15319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:03.827617 15319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:14:03.830259 15334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:03.831573 15319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:14:03.831769 15319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "ec0aaf2b787f4b59b05d2f2078542ba1"
format_stamp: "Formatted at 2026-09-28 05:13:57 on dist-test-slave-6718"
server_key: "51bfaf8659d21d40d0dcf5814235e4c8"
server_key_iv: "cc3ed5e701d457bb4e0ab2a5108c6e3c"
server_key_version: "encryptionkey@0"
I20260928 05:14:03.832724 15319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:03.856415 15319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:03.860152 15319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:03.860420 15319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:03.877841 15319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35525
I20260928 05:14:03.877885 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35525 every 8 connection(s)
I20260928 05:14:03.879341 15319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:14:03.880648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15319
I20260928 05:14:03.881631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38271
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35525
--webserver_port=42361
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:14:03.889210 15386 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:14:03.892534 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap starting.
I20260928 05:14:03.912847 15386 log.cc:824] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:03.928555 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:14:03.929112 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1: Bootstrap complete.
I20260928 05:14:03.938476 15386 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:03.939513 15386 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0aaf2b787f4b59b05d2f2078542ba1, State: Initialized, Role: FOLLOWER
I20260928 05:14:03.940074 15386 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:03.940323 15386 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:14:03.940456 15386 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:14:03.940626 15386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 05:14:03.944326 15386 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:03.944818 15386 leader_election.cc:302] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0aaf2b787f4b59b05d2f2078542ba1; no voters:
I20260928 05:14:03.945477 15386 leader_election.cc:288] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 05:14:03.945616 15390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 05:14:03.946861 15390 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [term 3 LEADER]: Becoming Leader. State: Replica: ec0aaf2b787f4b59b05d2f2078542ba1, State: Running, Role: LEADER
I20260928 05:14:03.947425 15390 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } }
I20260928 05:14:03.947696 15386 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:14:03.949849 15392 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0aaf2b787f4b59b05d2f2078542ba1. Latest consensus state: current_term: 3 leader_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } } }
I20260928 05:14:03.950223 15392 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:03.950907 15391 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0aaf2b787f4b59b05d2f2078542ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35525 } } }
I20260928 05:14:03.951261 15391 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0aaf2b787f4b59b05d2f2078542ba1 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:03.963527 15398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:14:03.972659 15398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:14:03.974167 15398 catalog_manager.cc:1259] Loaded cluster ID: db8e19ca0e0c4121bb5e384be1416289
I20260928 05:14:03.974344 15398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 05:14:03.976519 15398 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fed05e609ad google::LogMessage::Fail() at ??:0
@ 0x7fed05e6388a google::LogMessage::SendToLog() at ??:0
@ 0x7fed05e604de google::LogMessage::Flush() at ??:0
@ 0x7fed05e61a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fed1b893512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fed1b890976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fed1b918610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fed1b918439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fed08aac345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fed08aaf91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fed08aaf799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fed08a8aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fed0bd646db start_thread at ??:0
@ 0x7fed04cfa61f clone at ??:0
W20260928 05:14:04.273711 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:04.274003 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:04.274166 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:04.281486 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:04.281594 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:04.281769 15388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:14:04.291554 15388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42361
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:04.292797 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:04.294095 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:04.307082 15428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:04.307104 15427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:04.307086 15430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:04.308000 15388 server_base.cc:1034] running on GCE node
I20260928 05:14:04.308625 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:04.309567 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:04.310787 15388 hybrid_clock.cc:648] HybridClock initialized: now 1790572444310735 us; error 44 us; skew 500 ppm
I20260928 05:14:04.311105 15388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:04 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:04.314067 15388 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:14:04.315618 15388 webserver.cc:460] Webserver started at http://127.0.97.1:42361/ using document root <none> and password file <none>
I20260928 05:14:04.316286 15388 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:04.316427 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:04.321551 15388 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 05:14:04.324229 15437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:04.325529 15388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:14:04.325728 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "56bfc9ba215d4419a08fc3757806dc39"
format_stamp: "Formatted at 2026-09-28 05:13:58 on dist-test-slave-6718"
server_key: "8fb33f98c78c8749be4b606a7f138987"
server_key_iv: "286fa6f4b796ea9e9909ff3a8e5774f0"
server_key_version: "encryptionkey@0"
I20260928 05:14:04.326710 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:04.339068 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:04.342826 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:04.343086 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:04.343984 15388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:04.345221 15388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:04.345353 15388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:04.345477 15388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:04.345561 15388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:04.376677 15388 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38271
I20260928 05:14:04.376765 15549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38271 every 8 connection(s)
I20260928 05:14:04.378098 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:14:04.382826 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:04.380986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:37587)
0928 05:14:04.381690 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:04.381899 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
W20260928 05:14:04.383340 15550 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35525 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35525: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
I20260928 05:14:04.384074 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:04.383788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:55677)
0928 05:14:04.383917 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:04.383977 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:14:04.384821 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:04.384537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:56489)
0928 05:14:04.384644 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:04.384710 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 05:14:04.385052 15550 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:14:04.386941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15388
I20260928 05:14:04.387530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43551
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35525
--webserver_port=40327
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:04.709255 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:04.709533 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:04.709718 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:04.716991 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:04.717104 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:04.717283 15553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:14:04.726984 15553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40327
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:04.728289 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:04.729576 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:04.742853 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:04.743034 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:04.743009 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:04.743494 15553 server_base.cc:1034] running on GCE node
I20260928 05:14:04.744144 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:04.745057 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:04.746291 15553 hybrid_clock.cc:648] HybridClock initialized: now 1790572444746240 us; error 61 us; skew 500 ppm
I20260928 05:14:04.746670 15553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:04 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:04.749699 15553 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:14:04.751240 15553 webserver.cc:460] Webserver started at http://127.0.97.2:40327/ using document root <none> and password file <none>
I20260928 05:14:04.751912 15553 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:04.752060 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:04.757346 15553 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 05:14:04.760052 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:04.761428 15553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:14:04.761598 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "e75c30d5de58454c839d8cb85630294e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "eeb60f01854f62affd4bc8e50b516d31"
server_key_iv: "63bd7e5a6443b6016c3415c388d5ab40"
server_key_version: "encryptionkey@0"
I20260928 05:14:04.762596 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:04.785053 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:04.789100 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:04.789387 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:04.790391 15553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:04.791734 15553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:04.791859 15553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:04.791996 15553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:04.792083 15553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:04.825275 15553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43551
I20260928 05:14:04.825362 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43551 every 8 connection(s)
I20260928 05:14:04.826771 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:14:04.829187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15553
I20260928 05:14:04.829934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36475
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35525
--webserver_port=40683
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:14:04.832095 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:04.829994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:60563)
0928 05:14:04.830844 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:04.831177 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":327,"threads_started":1}
W20260928 05:14:04.832762 15681 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35525 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35525: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
I20260928 05:14:04.833602 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:04.833273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:46445)
0928 05:14:04.833424 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:04.833500 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:14:04.834517 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:04.834188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:33015)
0928 05:14:04.834333 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:04.834409 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 05:14:04.834805 15681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:14:05.146966 15684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:05.147225 15684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:05.147318 15684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:05.154426 15684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:05.154587 15684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:05.154727 15684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:14:05.164726 15684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40683
--tserver_master_addrs=127.0.97.62:35525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:05.166049 15684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:05.167374 15684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:05.180256 15690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:05.180392 15692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:05.180233 15689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:05.181182 15684 server_base.cc:1034] running on GCE node
I20260928 05:14:05.181746 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:05.182694 15684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:05.183898 15684 hybrid_clock.cc:648] HybridClock initialized: now 1790572445183841 us; error 49 us; skew 500 ppm
I20260928 05:14:05.184201 15684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:05 dist-test-slave-6718 krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:05.187323 15684 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:14:05.188694 15684 webserver.cc:460] Webserver started at http://127.0.97.3:40683/ using document root <none> and password file <none>
I20260928 05:14:05.189306 15684 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:05.189402 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:05.194454 15684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:14:05.197057 15699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:05.198387 15684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:14:05.198544 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "ba13fcdd7cc54c58a6bc3e5217bcbb8e"
format_stamp: "Formatted at 2026-09-28 05:13:59 on dist-test-slave-6718"
server_key: "012d88fd608b83b5bbfdb3d9f392986d"
server_key_iv: "7f33ee2ed15e2a8ec0455ba158e5ce22"
server_key_version: "encryptionkey@0"
I20260928 05:14:05.199512 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:05.223840 15684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:05.228238 15684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:05.228518 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:05.229489 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:05.230929 15684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:05.231076 15684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:05.231196 15684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:05.231292 15684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:05.264160 15684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36475
I20260928 05:14:05.264254 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36475 every 8 connection(s)
I20260928 05:14:05.265661 15684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:14:05.270449 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:05.268656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:56289)
0928 05:14:05.269357 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:05.269583 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":273,"threads_started":1}
W20260928 05:14:05.271112 15812 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35525 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35525: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
I20260928 05:14:05.271915 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:05.271608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:33515)
0928 05:14:05.271744 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:05.271812 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:14:05.272146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15684
I20260928 05:14:05.272794 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:05.272466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:51525)
0928 05:14:05.272603 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:05.272687 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:14:05.273114 15812 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:14:05.386281 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:05.385529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:44101)
0928 05:14:05.385973 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:05.386067 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260928 05:14:05.836045 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:05.835310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:34781)
0928 05:14:05.835739 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:05.835825 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260928 05:14:06.274436 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:06.273646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:49481)
0928 05:14:06.274110 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:06.274198 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260928 05:14:06.387753 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:06.387084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:44485)
0928 05:14:06.387513 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:06.387592 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":209,"threads_started":1}
I20260928 05:14:06.837548 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:06.836864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:33527)
0928 05:14:06.837312 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:06.837387 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":222,"threads_started":1}
I20260928 05:14:07.276084 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:07.275369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:42275)
0928 05:14:07.275836 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:07.275921 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260928 05:14:07.389439 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:07.388696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:51517)
0928 05:14:07.389164 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:07.389256 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20260928 05:14:07.839257 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:07.838520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:47763)
0928 05:14:07.839017 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:07.839095 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260928 05:14:08.277930 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:08.277123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:51569)
0928 05:14:08.277655 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:08.277742 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":310,"threads_started":1}
I20260928 05:14:08.391526 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:08.390744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:58313)
0928 05:14:08.391226 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:08.391326 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":239,"threads_started":1}
I20260928 05:14:08.841040 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:08.840227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:41797)
0928 05:14:08.840714 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:08.840833 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
I20260928 05:14:09.279804 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:09.279021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:40111)
0928 05:14:09.279480 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:09.279561 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260928 05:14:09.393393 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:09.392630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:42991)
0928 05:14:09.393124 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:09.393210 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
I20260928 05:14:09.842886 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:09.842071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:57071)
0928 05:14:09.842628 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:09.842706 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260928 05:14:10.281544 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:10.280785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:50925)
0928 05:14:10.281281 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:10.281367 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
I20260928 05:14:10.395130 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:10.394292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:35881)
0928 05:14:10.394872 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:10.394967 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":294,"threads_started":1}
I20260928 05:14:10.844509 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:10.843814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:60823)
0928 05:14:10.844243 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:10.844354 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260928 05:14:11.283275 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:11.282499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:52929)
0928 05:14:11.283006 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:11.283119 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":240,"threads_started":1}
I20260928 05:14:11.396793 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:11.396026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:46981)
0928 05:14:11.396499 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:11.396575 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":200,"threads_started":1}
I20260928 05:14:11.846194 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:11.845407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:39483)
0928 05:14:11.845902 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:11.845984 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":222,"threads_started":1}
I20260928 05:14:12.284989 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:12.284193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:57601)
0928 05:14:12.284689 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:12.284777 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":232,"threads_started":1}
I20260928 05:14:12.398480 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:12.397799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:43579)
0928 05:14:12.398224 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:12.398318 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260928 05:14:12.847780 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:12.847097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:44111)
0928 05:14:12.847547 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:12.847625 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260928 05:14:13.286504 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:13.285849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:55611)
0928 05:14:13.286271 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:13.286348 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":213,"threads_started":1}
I20260928 05:14:13.400141 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:13.399421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:56541)
0928 05:14:13.399894 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:13.399970 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":226,"threads_started":1}
I20260928 05:14:13.849300 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:13.848563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:36315)
0928 05:14:13.849046 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:13.849123 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
I20260928 05:14:14.288028 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:14.287336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:46861)
0928 05:14:14.287763 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:14.287862 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260928 05:14:14.401659 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:14.401000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:44155)
0928 05:14:14.401419 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:14.401500 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
I20260928 05:14:14.850984 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:14.850215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:57351)
0928 05:14:14.850736 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:14.850821 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":266,"threads_started":1}
I20260928 05:14:15.289639 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:15.288947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:59213)
0928 05:14:15.289404 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:15.289484 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":257,"threads_started":1}
I20260928 05:14:15.403467 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:15.402620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:58633)
0928 05:14:15.403150 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:15.403263 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":297,"threads_started":1}
I20260928 05:14:15.852522 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:15.851843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:52295)
0928 05:14:15.852248 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:15.852344 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":207,"threads_started":1}
I20260928 05:14:16.291440 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:16.290752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:42297)
0928 05:14:16.291166 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:16.291255 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260928 05:14:16.405010 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:16.404342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:47991)
0928 05:14:16.404770 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:16.404848 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20260928 05:14:16.854279 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:16.853517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:49479)
0928 05:14:16.854026 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:16.854121 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260928 05:14:17.293295 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:17.292421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:44907)
0928 05:14:17.293052 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:17.293146 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":346,"threads_started":1}
I20260928 05:14:17.406924 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:17.406077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:36649)
0928 05:14:17.406648 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:17.406735 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":270,"threads_started":1}
I20260928 05:14:17.856053 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:17.855288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:38581)
0928 05:14:17.855747 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:17.855861 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":223,"threads_started":1}
I20260928 05:14:18.295127 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:18.294310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:46547)
0928 05:14:18.294845 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:18.294943 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":295,"threads_started":1}
I20260928 05:14:18.408655 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:18.407954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:60651)
0928 05:14:18.408402 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:18.408479 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":207,"threads_started":1}
I20260928 05:14:18.857688 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:18.856935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:45053)
0928 05:14:18.857414 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:18.857508 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260928 05:14:19.296767 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:19.296028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:48479)
0928 05:14:19.296487 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:19.296574 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260928 05:14:19.410635 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:19.409811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.1:39333)
0928 05:14:19.410292 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:19.410383 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
I20260928 05:14:19.859719 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:19.858849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.2:55225)
0928 05:14:19.859389 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:19.859510 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":291,"threads_started":1}
I20260928 05:14:20.283596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15388
I20260928 05:14:20.298965 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:20.297886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35525 (local address 127.0.97.3:45185)
0928 05:14:20.298379 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:20.298689 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":266,"threads_started":1}
I20260928 05:14:20.304865 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:20.284878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.0.97.1:38271)
0928 05:14:20.285884 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:20.285907 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:20.285961 (+ 54us) server_negotiation.cc:407] Connection header received
0928 05:14:20.288380 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:20.288390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:20.288854 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:20.289249 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:20.290985 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:20.292311 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:20.293783 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:20.294577 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:20.294825 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:20.297802 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:20.297812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:20.297829 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:20.297870 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:20.301256 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:20.302068 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:20.302084 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:20.302093 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:20.302280 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:20.302768 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:20.302775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:20.302778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:20.303342 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:20.303722 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:20.304140 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 05:14:20.304581 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":636,"threads_started":1}
I20260928 05:14:20.306315 15545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:20.391410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15388
I20260928 05:14:20.408316 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15553
I20260928 05:14:20.426192 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:20.409401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.97.2:43551)
0928 05:14:20.410013 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:20.410028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:20.410065 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:14:20.411689 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:20.411700 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:20.412144 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:20.412451 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:20.413754 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:20.415116 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:20.416316 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:20.416981 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:20.417118 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:20.419875 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:20.419884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:20.419902 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:20.419941 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:20.422957 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:20.423670 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:20.423676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:20.423684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:20.423873 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:20.424247 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:20.424250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:20.424252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:20.424780 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:20.425007 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:20.425449 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 05:14:20.425897 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":356,"threads_started":1}
I20260928 05:14:20.427700 15676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:20.511219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15553
I20260928 05:14:20.529098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15684
I20260928 05:14:20.547385 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:20.530157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.97.3:36475)
0928 05:14:20.530843 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:20.530888 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:20.530931 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:14:20.532535 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:20.532544 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:20.532990 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:20.533279 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:20.534685 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:20.536011 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:20.537368 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:20.538080 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:20.538214 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:20.541078 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:20.541086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:20.541102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:20.541140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:20.544208 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:20.544867 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:20.544876 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:20.544883 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:20.545051 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:20.545451 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:20.545454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:20.545456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:20.545960 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:20.546168 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:20.546653 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 05:14:20.547087 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":305,"threads_started":1}
I20260928 05:14:20.548791 15807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:20.633389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15684
2026-09-28T05:14:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24416 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:14:20 dist-test-slave-6718 krb5kdc[15869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:14:20 dist-test-slave-6718 krb5kdc[15869](info): set up 2 sockets
Sep 28 05:14:20 dist-test-slave-6718 krb5kdc[15869](info): commencing operation
krb5kdc: starting...
W20260928 05:14:21.735100 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:14:21 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:14:21.964162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40623
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:22.298415 15882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:22.298839 15882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:22.299005 15882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:22.306295 15882 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:14:22.306457 15882 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:14:22.306546 15882 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:14:22.306624 15882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:14:22.306732 15882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:22.306849 15882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:22.306943 15882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:14:22.307042 15882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:14:22.317344 15882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:22.318696 15882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:22.320055 15882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:22.332167 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:22.332195 15890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:22.332252 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:22.333039 15882 server_base.cc:1034] running on GCE node
I20260928 05:14:22.333807 15882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:22.335256 15882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:22.336484 15882 hybrid_clock.cc:648] HybridClock initialized: now 1790572462336417 us; error 41 us; skew 500 ppm
I20260928 05:14:22.336838 15882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:22 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:22.344018 15882 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:14:22.345531 15882 webserver.cc:460] Webserver started at http://127.0.97.62:35825/ using document root <none> and password file <none>
I20260928 05:14:22.346259 15882 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:22.346623 15882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:22.346974 15882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:14:22.349973 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "afbc6586a5334ab495863402267d84f7"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "b09f4b48384fdc398cdc2c5491f87677"
server_key_iv: "20056251a37e0dac864c36254a5b3858"
server_key_version: "encryptionkey@0"
I20260928 05:14:22.350944 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "afbc6586a5334ab495863402267d84f7"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "b09f4b48384fdc398cdc2c5491f87677"
server_key_iv: "20056251a37e0dac864c36254a5b3858"
server_key_version: "encryptionkey@0"
I20260928 05:14:22.356345 15882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:14:22.363169 15897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:22.364832 15882 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260928 05:14:22.365031 15882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "afbc6586a5334ab495863402267d84f7"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "b09f4b48384fdc398cdc2c5491f87677"
server_key_iv: "20056251a37e0dac864c36254a5b3858"
server_key_version: "encryptionkey@0"
I20260928 05:14:22.365258 15882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:22.377785 15882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:22.382148 15882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:22.382417 15882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:22.401046 15882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40623
I20260928 05:14:22.401093 15948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40623 every 8 connection(s)
I20260928 05:14:22.402577 15882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:14:22.407608 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:22.410997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15882
I20260928 05:14:22.411207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:14:22.411960 388 external_mini_cluster.cc:1428] Setting key 9ab561621265f613a6f6067ebbd25c5d
I20260928 05:14:22.424628 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap starting.
I20260928 05:14:22.428399 15949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:22.429780 15949 log.cc:824] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:14:22 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:22.432991 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: No bootstrap required, opened a new log
I20260928 05:14:22.442317 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:22.416698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.97.62:40623)
0928 05:14:22.417572 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:22.417589 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:22.417645 (+ 56us) server_negotiation.cc:407] Connection header received
0928 05:14:22.420383 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:22.420427 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:22.421038 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:22.421557 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:22.423585 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:22.425869 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:22.427949 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:22.428726 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:22.428868 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:22.433490 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:22.433508 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:22.433530 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:22.433592 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:22.437652 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:22.438358 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:22.438367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:22.438375 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:22.438614 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:22.438969 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:22.438974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:22.438978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:22.439701 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:22.440022 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:22.440363 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 05:14:22.440824 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":324,"threads_started":1}
I20260928 05:14:22.444288 15949 raft_consensus.cc:348] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:22.444669 15949 raft_consensus.cc:374] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:22.444803 15949 raft_consensus.cc:729] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbc6586a5334ab495863402267d84f7, State: Initialized, Role: FOLLOWER
I20260928 05:14:22.445497 15949 consensus_queue.cc:260] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:22.445748 15949 raft_consensus.cc:388] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:14:22.445885 15949 raft_consensus.cc:482] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:14:22.446048 15949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:22.448557 15949 raft_consensus.cc:504] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:22.449061 15949 leader_election.cc:302] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afbc6586a5334ab495863402267d84f7; no voters:
I20260928 05:14:22.449628 15949 leader_election.cc:288] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:14:22.449764 15954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:14:22.450510 15954 raft_consensus.cc:686] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 LEADER]: Becoming Leader. State: Replica: afbc6586a5334ab495863402267d84f7, State: Running, Role: LEADER
I20260928 05:14:22.451025 15954 consensus_queue.cc:237] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:22.451736 15949 sys_catalog.cc:564] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:14:22.453807 15955 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afbc6586a5334ab495863402267d84f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } } }
I20260928 05:14:22.453706 15956 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afbc6586a5334ab495863402267d84f7. Latest consensus state: current_term: 1 leader_uuid: "afbc6586a5334ab495863402267d84f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } } }
I20260928 05:14:22.454191 15956 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:22.455505 15955 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:22.458918 15959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:14:22.463886 15959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:14:22.473111 15959 catalog_manager.cc:1347] Generated new cluster ID: 50f129f226cd481d86f69ca90a3c90ac
I20260928 05:14:22.473243 15959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:14:22.490139 15959 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:14:22.492022 15959 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:14:22.501868 15959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Generated new TSK 0
I20260928 05:14:22.502941 15959 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:14:22.558907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40623
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:22.882975 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:22.883257 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:22.883414 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:22.890774 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:22.890904 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:22.891094 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:14:22.900731 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:22.901963 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:22.903295 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:22.915657 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:22.915697 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:22.915720 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:22.916520 15975 server_base.cc:1034] running on GCE node
I20260928 05:14:22.917124 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:22.918040 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:22.919257 15975 hybrid_clock.cc:648] HybridClock initialized: now 1790572462919226 us; error 64 us; skew 500 ppm
I20260928 05:14:22.919569 15975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:22 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:22.922641 15975 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:14:22.924005 15975 webserver.cc:460] Webserver started at http://127.0.97.1:37807/ using document root <none> and password file <none>
I20260928 05:14:22.924681 15975 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:22.924821 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:22.925132 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:14:22.928033 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "fa13c62f66e64997a67b4152872da138"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "4b4ad92051923fb79ede7938893f9625"
server_key_iv: "d2ff00b8681868e9e0fe7131249beda0"
server_key_version: "encryptionkey@0"
I20260928 05:14:22.928792 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fa13c62f66e64997a67b4152872da138"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "4b4ad92051923fb79ede7938893f9625"
server_key_iv: "d2ff00b8681868e9e0fe7131249beda0"
server_key_version: "encryptionkey@0"
I20260928 05:14:22.934304 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:14:22.938037 15990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:22.939581 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:14:22.939787 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "fa13c62f66e64997a67b4152872da138"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "4b4ad92051923fb79ede7938893f9625"
server_key_iv: "d2ff00b8681868e9e0fe7131249beda0"
server_key_version: "encryptionkey@0"
I20260928 05:14:22.940013 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:22.985206 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:22.989271 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:22.989589 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:22.990597 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:22.991914 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:22.992017 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:22.992166 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:22.992242 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:23.028287 15975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35479
I20260928 05:14:23.028375 16102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35479 every 8 connection(s)
I20260928 05:14:23.029596 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:14:23.032130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 15975
I20260928 05:14:23.032366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:14:23.033002 388 external_mini_cluster.cc:1428] Setting key 6160f30a7bb8159db4f45312a315bc0f
Sep 28 05:14:23 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:23.051167 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:23.032276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38419 (local address 127.0.97.62:40623)
0928 05:14:23.034926 (+ 2650us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:23.034932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:23.034953 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:14:23.036276 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:23.036285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:23.036374 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:23.036496 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:23.037968 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.039142 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.040671 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.041398 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.041544 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:23.045532 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:23.045538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:23.045541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:23.045570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:23.048194 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:23.048904 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:23.048910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:23.048913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:23.049055 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:23.049478 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:23.049484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:23.049493 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:23.049745 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:23.049903 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:23.050595 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 05:14:23.050815 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2521,"thread_start_us":2381,"threads_started":1}
I20260928 05:14:23.052239 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:23.032975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:38419)
0928 05:14:23.033953 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:23.034008 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:23.036009 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:23.036686 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:23.036693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:23.037181 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:23.037788 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:23.037806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.039270 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.039274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:23.040527 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:23.040538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.041597 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.041601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:23.041785 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:23.042522 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:23.042579 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:23.045311 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:23.048344 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:23.048353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:23.048369 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:23.048784 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:23.049162 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:23.049168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:23.049172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:23.049359 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:23.049872 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:23.049879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:23.050206 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 05:14:23.050464 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 05:14:23.051005 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":409,"threads_started":1}
I20260928 05:14:23.053867 16103 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40623
I20260928 05:14:23.054167 16103 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:23.054895 16103 heartbeater.cc:511] Master 127.0.97.62:40623 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:14:23.056938 15914 ts_manager.cc:194] Registered new tserver with Master: fa13c62f66e64997a67b4152872da138 (127.0.97.1:35479)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:14:23.058408 15914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:14:23.085744 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40623
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:23.405536 16109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:23.405827 16109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:23.405992 16109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:23.412961 16109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:23.413097 16109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:23.413278 16109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:14:23.422505 16109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:23.423820 16109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:23.425105 16109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:23.437714 16117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:23.437680 16115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:23.438251 16114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:23.438791 16109 server_base.cc:1034] running on GCE node
I20260928 05:14:23.439430 16109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:23.440382 16109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:23.441615 16109 hybrid_clock.cc:648] HybridClock initialized: now 1790572463441570 us; error 57 us; skew 500 ppm
I20260928 05:14:23.441938 16109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:23 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:23.445341 16109 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:14:23.446790 16109 webserver.cc:460] Webserver started at http://127.0.97.2:45397/ using document root <none> and password file <none>
I20260928 05:14:23.447399 16109 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:23.447492 16109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:23.447741 16109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:14:23.450428 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "08be3a022fdf1a1dace982e4131f1726"
server_key_iv: "9d80bcc1567ae75a0467aeab21af37da"
server_key_version: "encryptionkey@0"
I20260928 05:14:23.451195 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "08be3a022fdf1a1dace982e4131f1726"
server_key_iv: "9d80bcc1567ae75a0467aeab21af37da"
server_key_version: "encryptionkey@0"
I20260928 05:14:23.456071 16109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:14:23.459446 16124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:23.460794 16109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:14:23.460927 16109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "08be3a022fdf1a1dace982e4131f1726"
server_key_iv: "9d80bcc1567ae75a0467aeab21af37da"
server_key_version: "encryptionkey@0"
I20260928 05:14:23.461095 16109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:23.510241 16109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:23.514845 16109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:23.515098 16109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:23.515992 16109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:23.517211 16109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:23.517287 16109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:23.517382 16109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:23.517490 16109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:23.550160 16109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38445
I20260928 05:14:23.550208 16236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38445 every 8 connection(s)
I20260928 05:14:23.551537 16109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:14:23.558490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16109
I20260928 05:14:23.558691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:14:23.559206 388 external_mini_cluster.cc:1428] Setting key 2294102805f5303786c3a8ce39353d0c
Sep 28 05:14:23 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:23.573819 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:23.554142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33777 (local address 127.0.97.62:40623)
0928 05:14:23.554598 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:23.554607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:23.555535 (+ 928us) server_negotiation.cc:407] Connection header received
0928 05:14:23.557560 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:23.557567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:23.557663 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:23.557790 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:23.559562 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.560546 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.562939 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.563329 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.563446 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:23.567621 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:23.567626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:23.567629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:23.567659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:23.570818 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:23.571505 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:23.571510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:23.571514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:23.571660 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:23.572090 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:23.572100 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:23.572102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:23.572337 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:23.572515 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:23.573324 (+ 809us) server_negotiation.cc:299] Negotiation successful
0928 05:14:23.573494 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":227,"threads_started":1}
I20260928 05:14:23.575265 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:23.554701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:33777)
0928 05:14:23.555360 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:23.555398 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:23.557262 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:23.558036 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:23.558044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:23.558712 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:23.559398 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:23.559418 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.560728 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.560732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:23.562780 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:23.562799 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:23.563580 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:23.563590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:23.563948 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:23.564680 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:23.564718 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:23.567385 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:23.570953 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:23.570962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:23.570977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:23.571376 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:23.571772 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:23.571778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:23.571781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:23.571969 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:23.572503 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:23.572514 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:23.572944 (+ 430us) client_negotiation.cc:769] Sending connection context
0928 05:14:23.573249 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 05:14:23.573816 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":262,"threads_started":1}
I20260928 05:14:23.576740 16237 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40623
I20260928 05:14:23.577049 16237 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:23.577745 16237 heartbeater.cc:511] Master 127.0.97.62:40623 requested a full tablet report, sending...
I20260928 05:14:23.579198 15914 ts_manager.cc:194] Registered new tserver with Master: 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
I20260928 05:14:23.580019 15914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33777
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:14:23.613323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40623
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:23.932937 16243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:23.933256 16243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:23.933387 16243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:23.940377 16243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:23.940500 16243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:23.940623 16243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:14:23.950314 16243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:23.951617 16243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:23.952817 16243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:23.965829 16248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:23.965714 16249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:23.965922 16251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:23.967312 16243 server_base.cc:1034] running on GCE node
I20260928 05:14:23.967896 16243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:23.968997 16243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:23.970234 16243 hybrid_clock.cc:648] HybridClock initialized: now 1790572463970141 us; error 52 us; skew 500 ppm
I20260928 05:14:23.970702 16243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:23 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:23.973826 16243 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:14:23.975355 16243 webserver.cc:460] Webserver started at http://127.0.97.3:44995/ using document root <none> and password file <none>
I20260928 05:14:23.976081 16243 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:23.976214 16243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:23.976537 16243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:14:23.979686 16243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "2e04e334ef59414f1c578c23e24c3d0e"
server_key_iv: "76342e89ab0916347efce16ba4c0d3fa"
server_key_version: "encryptionkey@0"
I20260928 05:14:23.980490 16243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "2e04e334ef59414f1c578c23e24c3d0e"
server_key_iv: "76342e89ab0916347efce16ba4c0d3fa"
server_key_version: "encryptionkey@0"
I20260928 05:14:23.985584 16243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:14:23.988991 16258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:23.990350 16243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:14:23.990623 16243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "2e04e334ef59414f1c578c23e24c3d0e"
server_key_iv: "76342e89ab0916347efce16ba4c0d3fa"
server_key_version: "encryptionkey@0"
I20260928 05:14:23.990833 16243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:24.007822 16243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:24.012033 16243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:24.012329 16243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:24.013298 16243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:24.014672 16243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:24.014778 16243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:24.014936 16243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:24.015010 16243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:24.050863 16243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35185
I20260928 05:14:24.050915 16370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35185 every 8 connection(s)
I20260928 05:14:24.052186 16243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:14:24.055640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16243
I20260928 05:14:24.055850 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:14:24.056463 388 external_mini_cluster.cc:1428] Setting key 042ec91ec5736b65367da609c8661724
I20260928 05:14:24.061367 16103 heartbeater.cc:503] Master 127.0.97.62:40623 was elected leader, sending a full tablet report...
Sep 28 05:14:24 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:24.073012 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.054855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37233 (local address 127.0.97.62:40623)
0928 05:14:24.055084 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.055090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.056383 (+ 1293us) server_negotiation.cc:407] Connection header received
0928 05:14:24.058491 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.058496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.058633 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.058746 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:24.060372 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.061101 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.062999 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.063436 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.063532 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.066975 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:24.066979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:24.066983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:24.067012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:24.069837 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.070825 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.070831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.070834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.071003 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.071476 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.071481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.071484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.071839 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:24.071951 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.072530 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.072687 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 05:14:24.073800 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.055356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:37233)
0928 05:14:24.056193 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.056251 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:24.058222 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:24.058976 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:24.058983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:24.059596 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:24.060206 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.060223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.061275 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.061283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.062867 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.062878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.063613 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.063617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.063797 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.064560 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:24.064581 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:24.066792 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:24.070021 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:24.070027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:24.070037 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:24.070705 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:24.071123 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:24.071129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:24.071132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:24.071345 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:24.071954 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:24.071960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:24.072231 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 05:14:24.072450 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 05:14:24.072783 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":315,"threads_started":1}
I20260928 05:14:24.075094 16371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40623
I20260928 05:14:24.075377 16371 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:24.076074 16371 heartbeater.cc:511] Master 127.0.97.62:40623 requested a full tablet report, sending...
I20260928 05:14:24.077342 15914 ts_manager.cc:194] Registered new tserver with Master: d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185)
I20260928 05:14:24.078047 15914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37233
I20260928 05:14:24.083653 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:14:24.100303 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.088111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.0.97.62:40623)
0928 05:14:24.088307 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.088315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.088737 (+ 422us) server_negotiation.cc:407] Connection header received
0928 05:14:24.088847 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.088851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.088904 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.088984 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:24.090266 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.091032 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.092338 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.092698 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.092783 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.095333 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:24.095338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:24.095341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:24.095370 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:24.097913 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.098624 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.098628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.098630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.098750 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.099075 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.099078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.099080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.099373 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:24.099561 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.099840 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.100003 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:14:24.104606 15914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:14:24.114998 15914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:14:24.145426 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.127180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.0.97.2:38445)
0928 05:14:24.136386 (+ 9206us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.136396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.136428 (+ 32us) server_negotiation.cc:407] Connection header received
0928 05:14:24.136546 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.136555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.136842 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.137075 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:24.137900 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.139596 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.142755 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.144191 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.144315 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.144512 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.144839 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.145006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9043,"thread_start_us":8917,"threads_started":1}
I20260928 05:14:24.145677 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.127069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38445 (local address 127.0.0.1:41852)
0928 05:14:24.127701 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.127748 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:24.127908 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:24.137083 (+ 9175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:24.137090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:24.137118 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:24.137685 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.137699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.139734 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.139743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.142565 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.142585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.144406 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.144412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.144603 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.144636 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:14:24.144769 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 05:14:24.145150 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":142,"thread_start_us":1157,"threads_started":1}
I20260928 05:14:24.145699 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.129452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.97.3:35185)
0928 05:14:24.135537 (+ 6085us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.135550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.136813 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 05:14:24.136963 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.136973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.137165 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.137330 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:24.138502 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.140026 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.142081 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.143416 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.143538 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.143767 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.145183 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.145351 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5856,"thread_start_us":5696,"threads_started":1}
I20260928 05:14:24.145933 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.128865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35185 (local address 127.0.0.1:48308)
0928 05:14:24.136683 (+ 7818us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.136699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:24.136820 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:24.137368 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:24.137375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:24.137406 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:24.138307 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.138318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.140167 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.140172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.141930 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.141942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.143915 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.143922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.144190 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.144252 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 05:14:24.144385 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 05:14:24.144588 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7716,"mutex_wait_us":61,"thread_start_us":7584,"threads_started":1}
I20260928 05:14:24.146306 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.127613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55766)
0928 05:14:24.136056 (+ 8443us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.136080 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:24.136351 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:24.137315 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:24.137320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:24.137345 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:24.137898 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.137906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.139590 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.139598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.142632 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.142646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.144854 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.144861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.145071 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.145104 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:14:24.145200 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 05:14:24.145309 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8282,"mutex_wait_us":210,"thread_start_us":8103,"threads_started":1}
I20260928 05:14:24.146363 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.128463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.0.97.1:35479)
0928 05:14:24.129141 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.129151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.136209 (+ 7058us) server_negotiation.cc:407] Connection header received
0928 05:14:24.136594 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.136603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.136858 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.137024 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:24.138084 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.139459 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.142805 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.144299 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.144478 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.144701 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.145824 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.145987 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":346,"threads_started":1}
I20260928 05:14:24.149533 16038 tablet_service.cc:1467] Processing CreateTablet for tablet d54803a008c14f3e87ec9ff9f29752ee (DEFAULT_TABLE table=test-table [id=50d02c0cf7b34e8cb5c3e5660e268cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:24.149534 16306 tablet_service.cc:1467] Processing CreateTablet for tablet d54803a008c14f3e87ec9ff9f29752ee (DEFAULT_TABLE table=test-table [id=50d02c0cf7b34e8cb5c3e5660e268cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:24.149534 16172 tablet_service.cc:1467] Processing CreateTablet for tablet d54803a008c14f3e87ec9ff9f29752ee (DEFAULT_TABLE table=test-table [id=50d02c0cf7b34e8cb5c3e5660e268cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:24.150877 16038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54803a008c14f3e87ec9ff9f29752ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:24.150924 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54803a008c14f3e87ec9ff9f29752ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:24.150918 16306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d54803a008c14f3e87ec9ff9f29752ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:24.162606 16389 tablet_bootstrap.cc:492] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Bootstrap starting.
I20260928 05:14:24.166370 16389 tablet_bootstrap.cc:654] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:24.166662 16391 tablet_bootstrap.cc:492] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Bootstrap starting.
I20260928 05:14:24.167960 16389 log.cc:824] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:24.168373 16390 tablet_bootstrap.cc:492] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Bootstrap starting.
I20260928 05:14:24.170678 16391 tablet_bootstrap.cc:654] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:24.171360 16390 tablet_bootstrap.cc:654] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:24.172232 16391 log.cc:824] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:24.172284 16390 log.cc:824] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:24.175647 16390 tablet_bootstrap.cc:492] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: No bootstrap required, opened a new log
I20260928 05:14:24.175653 16389 tablet_bootstrap.cc:492] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: No bootstrap required, opened a new log
I20260928 05:14:24.175659 16391 tablet_bootstrap.cc:492] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: No bootstrap required, opened a new log
I20260928 05:14:24.175927 16391 ts_tablet_manager.cc:1397] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 05:14:24.175922 16389 ts_tablet_manager.cc:1397] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260928 05:14:24.175927 16390 ts_tablet_manager.cc:1397] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 05:14:24.183117 16390 raft_consensus.cc:348] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.183487 16390 raft_consensus.cc:374] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:24.183586 16390 raft_consensus.cc:729] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c15b7a9e394e91b1defa2253f9a77f, State: Initialized, Role: FOLLOWER
I20260928 05:14:24.184167 16390 consensus_queue.cc:260] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.185257 16371 heartbeater.cc:503] Master 127.0.97.62:40623 was elected leader, sending a full tablet report...
I20260928 05:14:24.186008 16390 ts_tablet_manager.cc:1428] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260928 05:14:24.186725 16391 raft_consensus.cc:348] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.186721 16389 raft_consensus.cc:348] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.187134 16389 raft_consensus.cc:374] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:24.187134 16391 raft_consensus.cc:374] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:24.187234 16391 raft_consensus.cc:729] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcfe1bb5f3b4e0eae15c8191dde0dac, State: Initialized, Role: FOLLOWER
I20260928 05:14:24.187235 16389 raft_consensus.cc:729] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa13c62f66e64997a67b4152872da138, State: Initialized, Role: FOLLOWER
I20260928 05:14:24.187817 16391 consensus_queue.cc:260] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.187819 16389 consensus_queue.cc:260] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.189936 16237 heartbeater.cc:503] Master 127.0.97.62:40623 was elected leader, sending a full tablet report...
I20260928 05:14:24.190316 16391 ts_tablet_manager.cc:1428] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260928 05:14:24.190407 16389 ts_tablet_manager.cc:1428] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
W20260928 05:14:24.282001 16104 tablet.cc:2369] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:14:24.303635 16238 tablet.cc:2369] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:14:24.303921 16372 tablet.cc:2369] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:14:24.497184 16396 raft_consensus.cc:482] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:14:24.497608 16396 raft_consensus.cc:504] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.499382 16396 leader_election.cc:288] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445), d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185)
I20260928 05:14:24.506371 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.499817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33639 (local address 127.0.97.2:38445)
0928 05:14:24.500104 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.500111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.500472 (+ 361us) server_negotiation.cc:407] Connection header received
0928 05:14:24.500691 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.500699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.500791 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.500885 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:24.501595 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.502505 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.504498 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.505375 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.505484 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.505550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.505973 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.506107 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 05:14:24.506402 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.500728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45787 (local address 127.0.97.3:35185)
0928 05:14:24.500921 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.500929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.501185 (+ 256us) server_negotiation.cc:407] Connection header received
0928 05:14:24.501328 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.501334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.501416 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.501522 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:24.502109 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.503000 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.504711 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.505554 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.505677 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.505771 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.506052 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.506181 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:14:24.506472 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.499694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38445 (local address 127.0.97.1:33639)
0928 05:14:24.500341 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.500357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:24.500547 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:24.500990 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:24.500994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:24.501018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:24.501446 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.501462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.502638 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.502644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.504315 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.504327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.505603 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.505611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.505802 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.505837 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:14:24.505909 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 05:14:24.506032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":67,"spinlock_wait_cycles":23808,"thread_start_us":370,"threads_started":1}
I20260928 05:14:24.506822 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.500506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35185 (local address 127.0.97.1:45787)
0928 05:14:24.501084 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.501097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:24.501203 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:24.501569 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:24.501573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:24.501595 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:24.501964 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.501975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.503142 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.503146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.504555 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:24.504568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.505753 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.505758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:24.505863 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.505909 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 05:14:24.506007 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 05:14:24.506120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":262,"threads_started":1}
I20260928 05:14:24.507504 16192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54803a008c14f3e87ec9ff9f29752ee" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" is_pre_election: true
I20260928 05:14:24.507800 16326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54803a008c14f3e87ec9ff9f29752ee" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" is_pre_election: true
I20260928 05:14:24.507989 16192 raft_consensus.cc:2454] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 0.
I20260928 05:14:24.508251 16326 raft_consensus.cc:2454] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 0.
I20260928 05:14:24.508661 15991 leader_election.cc:302] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fcfe1bb5f3b4e0eae15c8191dde0dac, fa13c62f66e64997a67b4152872da138; no voters:
I20260928 05:14:24.508986 16396 raft_consensus.cc:2781] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:14:24.509104 16396 raft_consensus.cc:482] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:14:24.509253 16396 raft_consensus.cc:3037] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:24.511446 16396 raft_consensus.cc:504] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.512061 16396 leader_election.cc:288] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 election: Requested vote from peers 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445), d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185)
I20260928 05:14:24.512528 16192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54803a008c14f3e87ec9ff9f29752ee" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
I20260928 05:14:24.512604 16326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d54803a008c14f3e87ec9ff9f29752ee" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
I20260928 05:14:24.512748 16192 raft_consensus.cc:3037] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:24.512823 16326 raft_consensus.cc:3037] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:24.515026 16192 raft_consensus.cc:2454] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 1.
I20260928 05:14:24.515071 16326 raft_consensus.cc:2454] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 1.
I20260928 05:14:24.515407 15991 leader_election.cc:302] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fcfe1bb5f3b4e0eae15c8191dde0dac, fa13c62f66e64997a67b4152872da138; no voters:
I20260928 05:14:24.515692 16396 raft_consensus.cc:2781] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:14:24.516085 16396 raft_consensus.cc:686] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 1 LEADER]: Becoming Leader. State: Replica: fa13c62f66e64997a67b4152872da138, State: Running, Role: LEADER
I20260928 05:14:24.516542 16396 consensus_queue.cc:237] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } }
I20260928 05:14:24.520797 15913 catalog_manager.cc:5495] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa13c62f66e64997a67b4152872da138 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fa13c62f66e64997a67b4152872da138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } health_report { overall_health: HEALTHY } } }
I20260928 05:14:24.591238 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.585837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.0.97.1:35479)
0928 05:14:24.586169 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.586175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.586195 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:14:24.586367 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.586371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.586448 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.586636 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:14:24.587024 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.587750 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.589159 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.589547 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.589682 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.590150 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:14:24.590692 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:14:24.590856 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.590906 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.590983 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260928 05:14:24.599143 16192 raft_consensus.cc:1261] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Refusing update from remote peer fa13c62f66e64997a67b4152872da138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:14:24.599222 16326 raft_consensus.cc:1261] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Refusing update from remote peer fa13c62f66e64997a67b4152872da138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:14:24.600096 16400 consensus_queue.cc:1035] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:14:24.600499 16396 consensus_queue.cc:1035] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:14:24.613221 16405 mvcc.cc:205] Tried to move back new op lower bound from 7334184814988316672 to 7334184814663282688. Current Snapshot: MvccSnapshot[applied={T|T < 7334184814988316672}]
I20260928 05:14:24.625115 15914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:14:24.625399 15914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:14:24.628719 15914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Sending DeleteTablet for 3 replicas of tablet d54803a008c14f3e87ec9ff9f29752ee
I20260928 05:14:24.629918 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet d54803a008c14f3e87ec9ff9f29752ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41852
I20260928 05:14:24.629966 16306 tablet_service.cc:1514] Processing DeleteTablet for tablet d54803a008c14f3e87ec9ff9f29752ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48308
I20260928 05:14:24.630189 16038 tablet_service.cc:1514] Processing DeleteTablet for tablet d54803a008c14f3e87ec9ff9f29752ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55766
I20260928 05:14:24.630710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15975
I20260928 05:14:24.642335 16414 tablet_replica.cc:330] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: stopping tablet replica
I20260928 05:14:24.642977 16415 tablet_replica.cc:330] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: stopping tablet replica
I20260928 05:14:24.643605 16415 raft_consensus.cc:2229] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:14:24.644114 16414 raft_consensus.cc:2229] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:14:24.645293 16415 raft_consensus.cc:2258] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:24.645524 16414 raft_consensus.cc:2258] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:24.645685 16413 tablet_replica.cc:330] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: stopping tablet replica
I20260928 05:14:24.646433 16413 raft_consensus.cc:2229] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:14:24.647555 16413 raft_consensus.cc:2258] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:24.648365 16414 ts_tablet_manager.cc:1905] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:24.649648 16413 ts_tablet_manager.cc:1905] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:24.655690 16414 ts_tablet_manager.cc:1918] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:24.655925 16414 log.cc:1196] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/d54803a008c14f3e87ec9ff9f29752ee
Sep 28 05:14:24 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:14:24.656600 16414 ts_tablet_manager.cc:1939] T d54803a008c14f3e87ec9ff9f29752ee P d2c15b7a9e394e91b1defa2253f9a77f: Deleting consensus metadata
I20260928 05:14:24.656945 16413 ts_tablet_manager.cc:1918] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:24.657102 16413 log.cc:1196] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/d54803a008c14f3e87ec9ff9f29752ee
I20260928 05:14:24.657634 16413 ts_tablet_manager.cc:1939] T d54803a008c14f3e87ec9ff9f29752ee P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Deleting consensus metadata
I20260928 05:14:24.658718 15898 catalog_manager.cc:4841] TS d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185): tablet d54803a008c14f3e87ec9ff9f29752ee (table test-table [id=50d02c0cf7b34e8cb5c3e5660e268cf1]) successfully deleted
I20260928 05:14:24.660295 16415 ts_tablet_manager.cc:1905] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:24.660751 15898 catalog_manager.cc:4841] TS 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445): tablet d54803a008c14f3e87ec9ff9f29752ee (table test-table [id=50d02c0cf7b34e8cb5c3e5660e268cf1]) successfully deleted
I20260928 05:14:24.666023 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.632022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.97.1:35479)
0928 05:14:24.632232 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.632241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.644866 (+ 12625us) server_negotiation.cc:407] Connection header received
0928 05:14:24.645462 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.645468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.645554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.645633 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:24.647963 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.649155 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.651227 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.651920 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.652017 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.657712 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:24.657726 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:24.657744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:24.657798 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:24.662299 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.663213 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.663222 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.663226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.663411 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.663903 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.663911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.663917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.664495 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:24.664894 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.665229 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.665481 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:14:24.666760 16098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:24.666833 16415 ts_tablet_manager.cc:1918] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:24.667011 16415 log.cc:1196] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/d54803a008c14f3e87ec9ff9f29752ee
I20260928 05:14:24.765755 16415 ts_tablet_manager.cc:1939] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Deleting consensus metadata
I20260928 05:14:24.765816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15975
I20260928 05:14:24.784019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 16109
W20260928 05:14:24.784209 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35479 (error 108)
I20260928 05:14:24.786338 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.785514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55784)
0928 05:14:24.786063 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.786162 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":333,"threads_started":1}
W20260928 05:14:24.786583 15899 catalog_manager.cc:4584] TS fa13c62f66e64997a67b4152872da138 (127.0.97.1:35479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d54803a008c14f3e87ec9ff9f29752ee: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Sep 28 05:14:24 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:14:24.799471 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.784875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.0.97.2:38445)
0928 05:14:24.785060 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.785067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.785088 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:14:24.785350 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.785356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.785456 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.785555 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:24.787166 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.787901 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.789154 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.789535 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.789628 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.793739 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:24.793753 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:24.793777 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:24.793810 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:24.796851 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.797496 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.797503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.797505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.797641 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.798003 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.798006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.798008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.798352 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:24.798671 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.798915 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.799070 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:14:24.800164 16232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:24.840260 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.839752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55800)
0928 05:14:24.840018 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.840112 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 05:14:24.841305 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.840984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55810)
0928 05:14:24.841123 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.841187 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 05:14:24.890113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 16109
I20260928 05:14:24.901981 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.901379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55824)
0928 05:14:24.901640 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.901771 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 05:14:24.903436 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.902908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55836)
0928 05:14:24.903133 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.903232 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 05:14:24.908684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 16243
Sep 28 05:14:24 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:14:24.923576 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.909354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.97.3:35185)
0928 05:14:24.909515 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:24.909519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:24.909537 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:14:24.909667 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:24.909670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:24.909735 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:24.909818 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:24.911106 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.911817 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.913103 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:24.913478 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:24.913567 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:24.918009 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:24.918027 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:24.918042 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:24.918082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:24.921050 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.921783 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.921790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.921792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.921932 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:24.922229 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:24.922232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:24.922234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:24.922699 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:24.922967 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:24.923139 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 05:14:24.923296 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 05:14:24.924239 16366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:24.984175 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.983705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55852)
0928 05:14:24.983925 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.984022 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 05:14:24.985477 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:24.985021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:55858)
0928 05:14:24.985225 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:24.985315 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 05:14:25.014626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 16243
I20260928 05:14:25.032102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 15882
I20260928 05:14:25.033284 15944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:25.143412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 15882
I20260928 05:14:45.164892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40623
--webserver_interface=127.0.97.62
--webserver_port=35825
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:14:45.498378 16423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:45.498735 16423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:45.498925 16423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:45.505967 16423 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:14:45.506054 16423 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:14:45.506129 16423 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:14:45.506206 16423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:14:45.506302 16423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:45.506419 16423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:45.506495 16423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:14:45.506594 16423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:14:45.516999 16423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:45.518381 16423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:45.519786 16423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:45.542999 16432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:45.542987 16430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:45.543421 16429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:45.543910 16423 server_base.cc:1034] running on GCE node
I20260928 05:14:45.544788 16423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:45.546446 16423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:45.547719 16423 hybrid_clock.cc:648] HybridClock initialized: now 1790572485547650 us; error 46 us; skew 500 ppm
I20260928 05:14:45.548103 16423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:45 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:45.555874 16423 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:14:45.557766 16423 webserver.cc:460] Webserver started at http://127.0.97.62:35825/ using document root <none> and password file <none>
I20260928 05:14:45.558708 16423 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:45.558871 16423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:45.565539 16423 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 05:14:45.568717 16439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:45.570149 16423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:14:45.570343 16423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "afbc6586a5334ab495863402267d84f7"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "b09f4b48384fdc398cdc2c5491f87677"
server_key_iv: "20056251a37e0dac864c36254a5b3858"
server_key_version: "encryptionkey@0"
I20260928 05:14:45.571494 16423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:45.588214 16423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:45.592752 16423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:45.593056 16423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:45.612362 16423 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40623
I20260928 05:14:45.612396 16490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40623 every 8 connection(s)
I20260928 05:14:45.613909 16423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:14:45.618031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16423
I20260928 05:14:45.619057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35479
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40623
--webserver_port=37807
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:14:45.622555 16491 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:14:45.627981 16491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap starting.
I20260928 05:14:45.648276 16491 log.cc:824] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:45.660390 16491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:14:45.660969 16491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap complete.
I20260928 05:14:45.670269 16491 raft_consensus.cc:348] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:45.671197 16491 raft_consensus.cc:729] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbc6586a5334ab495863402267d84f7, State: Initialized, Role: FOLLOWER
I20260928 05:14:45.671804 16491 consensus_queue.cc:260] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:45.672070 16491 raft_consensus.cc:388] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:14:45.672205 16491 raft_consensus.cc:482] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:14:45.672379 16491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 05:14:45.676664 16491 raft_consensus.cc:504] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:45.677193 16491 leader_election.cc:302] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afbc6586a5334ab495863402267d84f7; no voters:
I20260928 05:14:45.677846 16491 leader_election.cc:288] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 05:14:45.678013 16495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 05:14:45.679255 16495 raft_consensus.cc:686] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 LEADER]: Becoming Leader. State: Replica: afbc6586a5334ab495863402267d84f7, State: Running, Role: LEADER
I20260928 05:14:45.679885 16495 consensus_queue.cc:237] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:14:45.680308 16491 sys_catalog.cc:564] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:14:45.682555 16496 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "afbc6586a5334ab495863402267d84f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } } }
I20260928 05:14:45.682929 16496 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:45.683019 16497 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afbc6586a5334ab495863402267d84f7. Latest consensus state: current_term: 2 leader_uuid: "afbc6586a5334ab495863402267d84f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } } }
I20260928 05:14:45.683351 16497 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:14:45.694156 16500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:14:45.702754 16500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:14:45.704190 16500 catalog_manager.cc:1259] Loaded cluster ID: 50f129f226cd481d86f69ca90a3c90ac
I20260928 05:14:45.704349 16500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:14:45.712967 16500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:14:45.714598 16500 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Loaded TSK: 0
I20260928 05:14:45.743257 16500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Generated new TSK 1
I20260928 05:14:45.743665 16500 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 05:14:45.972350 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:45.972659 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:45.972829 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:45.980512 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:45.980631 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:45.980845 16493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:14:45.991226 16493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37807
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:45.992581 16493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:45.993933 16493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:46.008004 16518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:46.008049 16519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:46.008772 16521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:46.009109 16493 server_base.cc:1034] running on GCE node
I20260928 05:14:46.009730 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:46.010768 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:46.011984 16493 hybrid_clock.cc:648] HybridClock initialized: now 1790572486011935 us; error 41 us; skew 500 ppm
I20260928 05:14:46.012295 16493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:46 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:46.015478 16493 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:14:46.019779 16493 webserver.cc:460] Webserver started at http://127.0.97.1:37807/ using document root <none> and password file <none>
I20260928 05:14:46.020488 16493 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:46.020643 16493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:46.026186 16493 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 05:14:46.029091 16528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:46.030581 16493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:14:46.030747 16493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "fa13c62f66e64997a67b4152872da138"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "4b4ad92051923fb79ede7938893f9625"
server_key_iv: "d2ff00b8681868e9e0fe7131249beda0"
server_key_version: "encryptionkey@0"
I20260928 05:14:46.031702 16493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:46.052205 16493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:46.056529 16493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:46.056807 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:46.057741 16493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:46.059495 16535 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 05:14:46.064606 16535 ts_tablet_manager.cc:1852] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 05:14:46.065546 16535 ts_tablet_manager.cc:1905] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:46.072525 16535 ts_tablet_manager.cc:1918] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:46.072747 16535 ts_tablet_manager.cc:1939] T d54803a008c14f3e87ec9ff9f29752ee P fa13c62f66e64997a67b4152872da138: Deleting consensus metadata
I20260928 05:14:46.073321 16493 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 05:14:46.073458 16493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260928 05:14:46.073596 16493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:46.073693 16493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:46.108675 16493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35479
I20260928 05:14:46.108907 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35479 every 8 connection(s)
I20260928 05:14:46.110038 16493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:14:46.115882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16493
I20260928 05:14:46.116505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38445
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40623
--webserver_port=45397
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:14:46 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:46.135932 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:46.113715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50223 (local address 127.0.97.62:40623)
0928 05:14:46.114279 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:46.114290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:46.115014 (+ 724us) server_negotiation.cc:407] Connection header received
0928 05:14:46.117334 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:46.117348 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:46.117691 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:46.118044 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:46.119657 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.120979 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.123004 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.123656 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.123802 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:46.128152 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:46.128161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:46.128176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:46.128212 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:46.131177 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:46.132137 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:46.132143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:46.132148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:46.132299 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:46.132785 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:46.132788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:46.132790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:46.133225 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:46.133447 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:46.134380 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 05:14:46.134820 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260928 05:14:46.136322 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:46.114047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:50223)
0928 05:14:46.114840 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:46.114881 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:46.116767 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:46.118246 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:46.118258 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:46.118913 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:46.119461 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:46.119479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.121152 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.121159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:46.122850 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:46.122868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.123895 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.123904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:46.124208 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:46.124913 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:46.124947 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:46.127872 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:46.131321 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:46.131333 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:46.131351 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:46.132003 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:46.132417 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:46.132424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:46.132431 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:46.132675 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:46.133383 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:46.133395 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:46.133875 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 05:14:46.134216 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 05:14:46.134874 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":299,"threads_started":1}
I20260928 05:14:46.139171 16642 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40623
I20260928 05:14:46.139520 16642 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:46.140445 16642 heartbeater.cc:511] Master 127.0.97.62:40623 requested a full tablet report, sending...
I20260928 05:14:46.142765 16456 ts_manager.cc:194] Registered new tserver with Master: fa13c62f66e64997a67b4152872da138 (127.0.97.1:35479)
I20260928 05:14:46.144356 16456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50223
W20260928 05:14:46.450230 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:46.450493 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:46.450697 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:46.457885 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:46.457983 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:46.458191 16646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:14:46.467796 16646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45397
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:46.469067 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:46.470358 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:46.483443 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:46.483531 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:46.483527 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:46.484356 16646 server_base.cc:1034] running on GCE node
I20260928 05:14:46.484946 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:46.485992 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:46.487187 16646 hybrid_clock.cc:648] HybridClock initialized: now 1790572486487141 us; error 36 us; skew 500 ppm
I20260928 05:14:46.487509 16646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:46 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:46.490497 16646 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:14:46.491989 16646 webserver.cc:460] Webserver started at http://127.0.97.2:45397/ using document root <none> and password file <none>
I20260928 05:14:46.492671 16646 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:46.492821 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:46.498189 16646 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:14:46.500837 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:46.502125 16646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:14:46.502298 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "08be3a022fdf1a1dace982e4131f1726"
server_key_iv: "9d80bcc1567ae75a0467aeab21af37da"
server_key_version: "encryptionkey@0"
I20260928 05:14:46.503330 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:46.535804 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:46.539772 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:46.540057 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:46.541021 16646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:46.542284 16646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:46.542416 16646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:46.542586 16646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:46.542688 16646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 05:14:46.574671 16646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38445
I20260928 05:14:46.574749 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38445 every 8 connection(s)
I20260928 05:14:46.576160 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:14:46.580983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16646
I20260928 05:14:46.581753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35185
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40623
--webserver_port=44995
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 05:14:46 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:46.598079 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:46.578491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54599 (local address 127.0.97.62:40623)
0928 05:14:46.578733 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:46.578739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:46.579950 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 05:14:46.582389 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:46.582393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:46.582460 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:46.582582 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:46.584575 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.585555 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.587523 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.587950 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.588097 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:46.592119 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:46.592125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:46.592129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:46.592161 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:46.594775 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:46.595737 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:46.595741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:46.595743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:46.595896 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:46.596387 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:46.596390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:46.596392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:46.596663 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:46.596772 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:46.597661 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 05:14:46.597838 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 05:14:46.599656 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:46.579095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:54599)
0928 05:14:46.579779 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:46.579819 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:46.582089 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:46.582838 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:46.582853 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:46.583625 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:46.584404 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:46.584433 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.585759 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.585767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:46.587375 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:46.587387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:46.588176 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:46.588183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:46.588422 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:46.589120 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:46.589156 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:46.591865 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:46.594915 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:46.594927 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:46.594945 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:46.595621 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:46.596004 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:46.596012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:46.596019 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:46.596276 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:46.596788 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:46.596799 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:46.597276 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 05:14:46.597573 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 05:14:46.598148 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":266,"threads_started":1}
I20260928 05:14:46.601173 16774 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40623
I20260928 05:14:46.601526 16774 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:46.602495 16774 heartbeater.cc:511] Master 127.0.97.62:40623 requested a full tablet report, sending...
I20260928 05:14:46.604138 16456 ts_manager.cc:194] Registered new tserver with Master: 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
I20260928 05:14:46.604921 16456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54599
W20260928 05:14:46.928083 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:14:46.928395 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:14:46.928568 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:14:46.935778 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:14:46.935928 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:14:46.936101 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:14:46.945770 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44995
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:14:46.947104 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:14:46.948370 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:14:46.960911 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:46.960960 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:14:46.960981 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:14:46.961683 16777 server_base.cc:1034] running on GCE node
I20260928 05:14:46.962272 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:14:46.963212 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:14:46.964434 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790572486964397 us; error 47 us; skew 500 ppm
I20260928 05:14:46.964761 16777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:14:46 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:14:46.967931 16777 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:14:46.969331 16777 webserver.cc:460] Webserver started at http://127.0.97.3:44995/ using document root <none> and password file <none>
I20260928 05:14:46.970048 16777 fs_manager.cc:359] Metadata directory not provided
I20260928 05:14:46.970206 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:14:46.975643 16777 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:14:46.978252 16792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:46.979652 16777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:14:46.979837 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "2e04e334ef59414f1c578c23e24c3d0e"
server_key_iv: "76342e89ab0916347efce16ba4c0d3fa"
server_key_version: "encryptionkey@0"
I20260928 05:14:46.980942 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:14:47.025573 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:14:47.030074 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:14:47.030352 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:14:47.031407 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:14:47.032912 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:14:47.033041 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:47.033167 16777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:14:47.033253 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:14:47.065923 16777 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35185
I20260928 05:14:47.066010 16904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35185 every 8 connection(s)
I20260928 05:14:47.067361 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:14:47.069312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16777
Sep 28 05:14:47 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:14:47.084757 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.070677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.97.62:40623)
0928 05:14:47.071176 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.071181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.071195 (+ 14us) server_negotiation.cc:407] Connection header received
0928 05:14:47.071244 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.071248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.071309 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.071417 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:47.072969 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.073736 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.075222 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.075814 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.075944 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.078776 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:47.078784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:47.078788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:47.078826 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:47.081873 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.082628 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.082632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.082634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.082761 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.083204 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.083207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.083209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.083500 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:47.083697 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.084104 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.084304 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
I20260928 05:14:47.088480 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.070129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46065 (local address 127.0.97.62:40623)
0928 05:14:47.070332 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.070338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.071772 (+ 1434us) server_negotiation.cc:407] Connection header received
0928 05:14:47.073588 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.073593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.073663 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.073763 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:47.075285 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.076121 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.077928 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.078507 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.078679 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.082321 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:47.082330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:47.082336 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:47.082374 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:47.085583 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.086433 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.086439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.086464 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.086649 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.087009 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.087014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.087016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.087344 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:47.087468 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.088133 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.088266 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:14:47.089432 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.070724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:46065)
0928 05:14:47.071602 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.071672 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:47.073306 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:47.073992 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:47.074000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:47.074465 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:14:47.075120 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.075145 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.076276 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.076282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.077790 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.077810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.078772 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.078780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.079017 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.079652 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:14:47.079683 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:14:47.082143 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:14:47.085700 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:47.085707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:47.085722 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:47.086285 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:47.086739 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:14:47.086742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:14:47.086744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:14:47.086905 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:14:47.087492 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:14:47.087500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:14:47.087832 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 05:14:47.088048 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 05:14:47.088378 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":341,"threads_started":1}
I20260928 05:14:47.090806 16905 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40623
I20260928 05:14:47.091111 16905 heartbeater.cc:464] Registering TS with master...
I20260928 05:14:47.091771 16905 heartbeater.cc:511] Master 127.0.97.62:40623 requested a full tablet report, sending...
I20260928 05:14:47.093093 16456 ts_manager.cc:194] Registered new tserver with Master: d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185)
I20260928 05:14:47.093892 16456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46065
I20260928 05:14:47.099565 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:14:47.105576 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.100485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.0.97.62:40623)
0928 05:14:47.100665 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.100674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.101119 (+ 445us) server_negotiation.cc:407] Connection header received
0928 05:14:47.101283 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.101286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.101424 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.101532 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:14:47.101888 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.102665 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.104061 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.104446 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.104531 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.104957 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:14:47.105161 (+ 204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 05:14:47.105408 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42476: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 05:14:47.105712 16644 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42476: token expired
W20260928 05:14:47.106415 16377 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:40623)
I20260928 05:14:47.119984 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.106883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.0.97.62:40623)
0928 05:14:47.107185 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.107190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.107208 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:14:47.107250 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.107254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.107346 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.107438 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:47.109247 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.110290 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.112116 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.112734 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.112864 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.115357 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:47.115362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:47.115365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:47.115395 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:47.117682 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.118263 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.118267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.118269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.118376 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.118754 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.118757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.118759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.119074 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:47.119253 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.119495 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.119669 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:14:47.123219 16456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:14:47.130151 16456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:14:47.148006 16642 heartbeater.cc:503] Master 127.0.97.62:40623 was elected leader, sending a full tablet report...
I20260928 05:14:47.156530 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.141191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35185 (local address 127.0.0.1:51208)
0928 05:14:47.142141 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.142181 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:47.142346 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:47.144019 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:47.144029 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:47.144071 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:47.145874 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.145895 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.149799 (+ 3904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.149806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.153580 (+ 3774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.153596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.155550 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.155560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.155762 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.155824 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 05:14:47.155932 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:14:47.156137 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":526,"threads_started":1}
I20260928 05:14:47.157253 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.142124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.0.97.3:35185)
0928 05:14:47.142924 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.142935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.142974 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:14:47.143105 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.143116 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.143399 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.143638 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:47.146162 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.149576 (+ 3414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.153734 (+ 4158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.155280 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.156385 (+ 1105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.156648 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.156722 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.156876 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":445,"threads_started":1}
I20260928 05:14:47.161644 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.142557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38445 (local address 127.0.0.1:58138)
0928 05:14:47.153887 (+ 11330us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.153912 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:47.154023 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:47.155036 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:47.155042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:47.155066 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:47.155458 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.155476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.157540 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.157544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.159544 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.159556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.160859 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.160866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.161038 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.161061 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:14:47.161185 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 05:14:47.161316 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11156,"thread_start_us":11003,"threads_started":1}
I20260928 05:14:47.161123 16840 tablet_service.cc:1467] Processing CreateTablet for tablet 3d458948ba7d4895aac49c043ecdfb3a (DEFAULT_TABLE table=test-table [id=0fc48a25adb640bb9731ce8ec16b9fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:47.161960 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.143010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.97.2:38445)
0928 05:14:47.154173 (+ 11163us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.154182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.154213 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:14:47.154313 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.154322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.154608 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.154812 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:47.155665 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.157115 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.159718 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.160635 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.160778 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.160966 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.161234 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.161384 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11000,"thread_start_us":10858,"threads_started":1}
I20260928 05:14:47.162717 16840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d458948ba7d4895aac49c043ecdfb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:47.164340 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.142958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.0.97.1:35479)
0928 05:14:47.155476 (+ 12518us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.155485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.155512 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:14:47.155597 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.155606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.155849 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.156029 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:47.156873 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.159110 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.161477 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.162917 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.163052 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.163801 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.163869 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.164022 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11443,"thread_start_us":11323,"threads_started":1}
I20260928 05:14:47.164651 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.141588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58212)
0928 05:14:47.152661 (+ 11073us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.152687 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:47.152795 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:47.156034 (+ 3239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:47.156039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:47.156062 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:47.156417 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.156429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.159261 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.159266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.161290 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.161304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.163125 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.163132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.163302 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.163334 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:14:47.163427 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:14:47.163564 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10929,"thread_start_us":11279,"threads_started":1}
I20260928 05:14:47.165861 16709 tablet_service.cc:1467] Processing CreateTablet for tablet 3d458948ba7d4895aac49c043ecdfb3a (DEFAULT_TABLE table=test-table [id=0fc48a25adb640bb9731ce8ec16b9fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:47.166519 16577 tablet_service.cc:1467] Processing CreateTablet for tablet 3d458948ba7d4895aac49c043ecdfb3a (DEFAULT_TABLE table=test-table [id=0fc48a25adb640bb9731ce8ec16b9fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:14:47.167249 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d458948ba7d4895aac49c043ecdfb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:47.167783 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d458948ba7d4895aac49c043ecdfb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:14:47.175015 16921 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Bootstrap starting.
I20260928 05:14:47.177353 16922 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Bootstrap starting.
I20260928 05:14:47.178781 16921 tablet_bootstrap.cc:654] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:47.180238 16921 log.cc:824] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:47.180922 16922 tablet_bootstrap.cc:654] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:47.182188 16922 log.cc:824] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:47.182657 16924 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Bootstrap starting.
I20260928 05:14:47.183377 16921 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: No bootstrap required, opened a new log
I20260928 05:14:47.183670 16921 ts_tablet_manager.cc:1397] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 05:14:47.185292 16922 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: No bootstrap required, opened a new log
I20260928 05:14:47.185531 16922 ts_tablet_manager.cc:1397] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 05:14:47.185814 16924 tablet_bootstrap.cc:654] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Neither blocks nor log segments found. Creating new log.
I20260928 05:14:47.186874 16924 log.cc:824] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Log is configured to *not* fsync() on all Append() calls
I20260928 05:14:47.189131 16924 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: No bootstrap required, opened a new log
I20260928 05:14:47.189445 16924 ts_tablet_manager.cc:1397] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 05:14:47.194463 16921 raft_consensus.cc:348] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.194938 16921 raft_consensus.cc:374] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:47.195036 16921 raft_consensus.cc:729] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c15b7a9e394e91b1defa2253f9a77f, State: Initialized, Role: FOLLOWER
I20260928 05:14:47.195680 16921 consensus_queue.cc:260] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.196048 16922 raft_consensus.cc:348] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.196506 16922 raft_consensus.cc:374] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:47.196606 16922 raft_consensus.cc:729] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa13c62f66e64997a67b4152872da138, State: Initialized, Role: FOLLOWER
I20260928 05:14:47.196961 16905 heartbeater.cc:503] Master 127.0.97.62:40623 was elected leader, sending a full tablet report...
I20260928 05:14:47.197214 16922 consensus_queue.cc:260] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.197454 16924 raft_consensus.cc:348] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.197877 16924 raft_consensus.cc:374] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:14:47.197984 16924 raft_consensus.cc:729] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcfe1bb5f3b4e0eae15c8191dde0dac, State: Initialized, Role: FOLLOWER
I20260928 05:14:47.198638 16924 consensus_queue.cc:260] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.198945 16922 ts_tablet_manager.cc:1428] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260928 05:14:47.200047 16921 ts_tablet_manager.cc:1428] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260928 05:14:47.200881 16774 heartbeater.cc:503] Master 127.0.97.62:40623 was elected leader, sending a full tablet report...
I20260928 05:14:47.201344 16924 ts_tablet_manager.cc:1428] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 05:14:47.248124 16928 raft_consensus.cc:482] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:14:47.248446 16928 raft_consensus.cc:504] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.250063 16928 leader_election.cc:288] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185), 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
I20260928 05:14:47.257840 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.250421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55549 (local address 127.0.97.3:35185)
0928 05:14:47.250632 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.250636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.251235 (+ 599us) server_negotiation.cc:407] Connection header received
0928 05:14:47.251443 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.251452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.251569 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.251672 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:47.252280 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.253305 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.255749 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.256931 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.257059 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.257163 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.257483 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.257585 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:14:47.257932 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.250421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35185 (local address 127.0.97.1:55549)
0928 05:14:47.251080 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.251094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:47.251271 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:47.251707 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:47.251713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:47.251740 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:47.252136 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.252146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.253438 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.253441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.255605 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.255619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.257130 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.257135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.257300 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.257354 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 05:14:47.257448 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:14:47.257577 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":368,"threads_started":1}
I20260928 05:14:47.258119 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.251370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59365 (local address 127.0.97.2:38445)
0928 05:14:47.251585 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.251591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.251921 (+ 330us) server_negotiation.cc:407] Connection header received
0928 05:14:47.252085 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.252089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.252145 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.252240 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:14:47.252730 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.253651 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.255949 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.257163 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.257281 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.257345 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.257716 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.257840 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:14:47.258644 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.251196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38445 (local address 127.0.97.1:59365)
0928 05:14:47.251804 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.251822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:14:47.251952 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:14:47.252273 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:14:47.252280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:14:47.252309 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:14:47.252573 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.252586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.253768 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.253773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.255815 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:14:47.255824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.257376 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.257382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:14:47.257550 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.257585 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:14:47.257665 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 05:14:47.257791 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":303,"threads_started":1}
I20260928 05:14:47.258977 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d458948ba7d4895aac49c043ecdfb3a" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" is_pre_election: true
I20260928 05:14:47.259542 16860 raft_consensus.cc:2454] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 0.
I20260928 05:14:47.259580 16729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d458948ba7d4895aac49c043ecdfb3a" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" is_pre_election: true
I20260928 05:14:47.260129 16729 raft_consensus.cc:2454] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 0.
I20260928 05:14:47.260237 16529 leader_election.cc:302] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2c15b7a9e394e91b1defa2253f9a77f, fa13c62f66e64997a67b4152872da138; no voters:
I20260928 05:14:47.260640 16928 raft_consensus.cc:2781] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:14:47.260792 16928 raft_consensus.cc:482] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:14:47.260895 16928 raft_consensus.cc:3037] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:47.263361 16928 raft_consensus.cc:504] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.264035 16928 leader_election.cc:288] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 election: Requested vote from peers d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185), 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
I20260928 05:14:47.264463 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d458948ba7d4895aac49c043ecdfb3a" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
I20260928 05:14:47.264546 16729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d458948ba7d4895aac49c043ecdfb3a" candidate_uuid: "fa13c62f66e64997a67b4152872da138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
I20260928 05:14:47.264683 16860 raft_consensus.cc:3037] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:47.264734 16729 raft_consensus.cc:3037] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:14:47.267967 16729 raft_consensus.cc:2454] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 1.
I20260928 05:14:47.267967 16860 raft_consensus.cc:2454] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa13c62f66e64997a67b4152872da138 in term 1.
I20260928 05:14:47.268356 16529 leader_election.cc:302] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2c15b7a9e394e91b1defa2253f9a77f, fa13c62f66e64997a67b4152872da138; no voters:
I20260928 05:14:47.268615 16928 raft_consensus.cc:2781] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:14:47.269042 16928 raft_consensus.cc:686] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 LEADER]: Becoming Leader. State: Replica: fa13c62f66e64997a67b4152872da138, State: Running, Role: LEADER
I20260928 05:14:47.269508 16928 consensus_queue.cc:237] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:14:47.274209 16454 catalog_manager.cc:5495] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa13c62f66e64997a67b4152872da138 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fa13c62f66e64997a67b4152872da138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } health_report { overall_health: UNKNOWN } } }
I20260928 05:14:47.304589 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.299299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.0.97.1:35479)
0928 05:14:47.299515 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.299520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.299540 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:14:47.299737 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.299741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.299816 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.299986 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:14:47.300342 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.301067 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.302437 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.302907 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.303051 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.303546 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:14:47.304015 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:14:47.304191 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.304243 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.304320 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:14:47.313150 16729 raft_consensus.cc:1261] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Refusing update from remote peer fa13c62f66e64997a67b4152872da138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:14:47.313150 16860 raft_consensus.cc:1261] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Refusing update from remote peer fa13c62f66e64997a67b4152872da138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:14:47.314194 16928 consensus_queue.cc:1035] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:14:47.314592 16932 consensus_queue.cc:1035] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:14:47.318933 16906 tablet.cc:2369] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:14:47.340092 16455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:14:47.340333 16455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:14:47.343924 16455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Sending DeleteTablet for 3 replicas of tablet 3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:14:47.345121 16577 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d458948ba7d4895aac49c043ecdfb3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58212
I20260928 05:14:47.345249 16840 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d458948ba7d4895aac49c043ecdfb3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51208
I20260928 05:14:47.345898 16709 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d458948ba7d4895aac49c043ecdfb3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:14:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58138
I20260928 05:14:47.346277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 16493
I20260928 05:14:47.357745 16943 tablet_replica.cc:330] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: stopping tablet replica
I20260928 05:14:47.358338 16943 raft_consensus.cc:2229] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:14:47.359174 16943 raft_consensus.cc:2258] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:47.360095 16944 tablet_replica.cc:330] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: stopping tablet replica
I20260928 05:14:47.360644 16944 raft_consensus.cc:2229] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:14:47.361264 16944 raft_consensus.cc:2258] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:47.362126 16945 tablet_replica.cc:330] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: stopping tablet replica
I20260928 05:14:47.362514 16945 raft_consensus.cc:2229] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:14:47.363005 16945 raft_consensus.cc:2258] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:14:47.364514 16944 ts_tablet_manager.cc:1905] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:47.364861 16945 ts_tablet_manager.cc:1905] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:47.365734 16943 ts_tablet_manager.cc:1905] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:14:47.368520 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.347065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.0.97.1:35479)
0928 05:14:47.347279 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.347288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.347348 (+ 60us) server_negotiation.cc:407] Connection header received
0928 05:14:47.347546 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.347554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.347659 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.347812 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:47.349534 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.350846 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.352815 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.353494 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.353611 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.357323 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:47.357339 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:47.357362 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:47.357406 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:47.364000 (+ 6594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.365099 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.365108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.365112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.365301 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.365763 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.365768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.365771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.366296 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:47.366627 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.367886 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.368130 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:14:47.369494 16637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:47.372066 16945 ts_tablet_manager.cc:1918] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:47.372267 16945 log.cc:1196] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:14:47.372954 16945 ts_tablet_manager.cc:1939] T 3d458948ba7d4895aac49c043ecdfb3a P 9fcfe1bb5f3b4e0eae15c8191dde0dac: Deleting consensus metadata
I20260928 05:14:47.374840 16944 ts_tablet_manager.cc:1918] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:14:47.375023 16944 log.cc:1196] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:14:47.374987 16440 catalog_manager.cc:4841] TS 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445): tablet 3d458948ba7d4895aac49c043ecdfb3a (table test-table [id=0fc48a25adb640bb9731ce8ec16b9fe5]) successfully deleted
I20260928 05:14:47.375466 16944 ts_tablet_manager.cc:1939] T 3d458948ba7d4895aac49c043ecdfb3a P d2c15b7a9e394e91b1defa2253f9a77f: Deleting consensus metadata
I20260928 05:14:47.376888 16440 catalog_manager.cc:4841] TS d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185): tablet 3d458948ba7d4895aac49c043ecdfb3a (table test-table [id=0fc48a25adb640bb9731ce8ec16b9fe5]) successfully deleted
I20260928 05:14:47.466730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 16493
I20260928 05:14:47.485827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 16646
W20260928 05:14:47.485898 16441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35479 (error 108)
I20260928 05:14:47.487890 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.487400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58228)
0928 05:14:47.487595 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.487713 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 05:14:47.488184 16441 catalog_manager.cc:4584] TS fa13c62f66e64997a67b4152872da138 (127.0.97.1:35479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d458948ba7d4895aac49c043ecdfb3a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
I20260928 05:14:47.499504 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.486680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.0.97.2:38445)
0928 05:14:47.486902 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.486907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.487021 (+ 114us) server_negotiation.cc:407] Connection header received
0928 05:14:47.487245 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.487254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.487358 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.487471 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:47.488867 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.489578 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.490792 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.491161 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.491255 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.493817 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:47.493829 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:47.493844 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:47.493870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:47.496702 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.497328 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.497334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.497336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.497448 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.497747 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.497750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.497752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.498286 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:47.498518 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.498850 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.499216 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 05:14:47.500139 16769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:47.541762 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.541196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58230)
0928 05:14:47.541431 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.541566 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 05:14:47.542850 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.542430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58246)
0928 05:14:47.542616 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.542699 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:14:47.589531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 16646
I20260928 05:14:47.603494 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.602918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58248)
0928 05:14:47.603169 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.603287 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 05:14:47.604625 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.604191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58256)
0928 05:14:47.604365 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.604461 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 05:14:47.607342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 16777
I20260928 05:14:47.621233 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.608110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.0.97.3:35185)
0928 05:14:47.608312 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:14:47.608317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:14:47.608335 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:14:47.608435 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:14:47.608439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:14:47.608519 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:14:47.608604 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:14:47.609902 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.610762 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.612163 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:14:47.612548 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:14:47.612640 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:14:47.615729 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:14:47.615744 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:14:47.615760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:14:47.615789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:14:47.618501 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.619318 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.619325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.619327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.619461 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:14:47.619823 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:14:47.619826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:14:47.619828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:14:47.620168 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:14:47.620522 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 05:14:47.620787 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 05:14:47.620952 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 05:14:47.621865 16900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:47.685326 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.684833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58264)
0928 05:14:47.685036 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.685184 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 05:14:47.686442 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:14:47.686013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35479 (local address 127.0.0.1:58278)
0928 05:14:47.686263 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 05:14:47.686339 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 05:14:47.713786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 16777
I20260928 05:14:47.730016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 16423
I20260928 05:14:47.731248 16486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:14:47.844220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 16423
I20260928 05:15:07.870019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40623
--webserver_interface=127.0.97.62
--webserver_port=35825
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:08.211905 16951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:08.212271 16951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:08.212433 16951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:08.219661 16951 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 05:15:08.219815 16951 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 05:15:08.219888 16951 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 05:15:08.219969 16951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:08.220050 16951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:08.220191 16951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:08.220287 16951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:08.220366 16951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:08.230341 16951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:08.231739 16951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:08.233114 16951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:08.245201 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:08.245332 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:08.245379 16959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:08.245770 16951 server_base.cc:1034] running on GCE node
I20260928 05:15:08.246701 16951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:08.248203 16951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:08.249461 16951 hybrid_clock.cc:648] HybridClock initialized: now 1790572508249402 us; error 40 us; skew 500 ppm
I20260928 05:15:08.249795 16951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:08 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:08.253162 16951 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:15:08.254845 16951 webserver.cc:460] Webserver started at http://127.0.97.62:35825/ using document root <none> and password file <none>
I20260928 05:15:08.255560 16951 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:08.255689 16951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:08.261184 16951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 05:15:08.264047 16966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:08.265388 16951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:15:08.265565 16951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "afbc6586a5334ab495863402267d84f7"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "b09f4b48384fdc398cdc2c5491f87677"
server_key_iv: "20056251a37e0dac864c36254a5b3858"
server_key_version: "encryptionkey@0"
I20260928 05:15:08.266527 16951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:08.292973 16951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:08.297642 16951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:08.297946 16951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:08.317258 16951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40623
I20260928 05:15:08.317305 17017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40623 every 8 connection(s)
I20260928 05:15:08.318851 16951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:08.324793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 16951
I20260928 05:15:08.325948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35479
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40623
--webserver_port=37807
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:15:08.327702 17018 sys_catalog.cc:263] Verifying existing consensus state
I20260928 05:15:08.333631 17018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap starting.
I20260928 05:15:08.357568 17018 log.cc:824] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:08.376577 17018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:15:08.377228 17018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Bootstrap complete.
I20260928 05:15:08.387727 17018 raft_consensus.cc:348] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:15:08.388721 17018 raft_consensus.cc:729] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbc6586a5334ab495863402267d84f7, State: Initialized, Role: FOLLOWER
I20260928 05:15:08.389433 17018 consensus_queue.cc:260] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:15:08.389684 17018 raft_consensus.cc:388] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:08.389815 17018 raft_consensus.cc:482] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:08.390023 17018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 2 FOLLOWER]: Advancing to term 3
I20260928 05:15:08.394279 17018 raft_consensus.cc:504] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:15:08.394922 17018 leader_election.cc:302] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afbc6586a5334ab495863402267d84f7; no voters:
I20260928 05:15:08.395695 17018 leader_election.cc:288] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 05:15:08.395852 17022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 05:15:08.397142 17022 raft_consensus.cc:686] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [term 3 LEADER]: Becoming Leader. State: Replica: afbc6586a5334ab495863402267d84f7, State: Running, Role: LEADER
I20260928 05:15:08.397822 17022 consensus_queue.cc:237] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } }
I20260928 05:15:08.398109 17018 sys_catalog.cc:564] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:08.400457 17024 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afbc6586a5334ab495863402267d84f7. Latest consensus state: current_term: 3 leader_uuid: "afbc6586a5334ab495863402267d84f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } } }
I20260928 05:15:08.400837 17024 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:08.400818 17023 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "afbc6586a5334ab495863402267d84f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbc6586a5334ab495863402267d84f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40623 } } }
I20260928 05:15:08.401372 17023 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:08.406975 17029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:08.422411 17029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:08.423993 17029 catalog_manager.cc:1259] Loaded cluster ID: 50f129f226cd481d86f69ca90a3c90ac
I20260928 05:15:08.424206 17029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:08.428553 17029 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:08.430438 17029 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Loaded TSK: 0
I20260928 05:15:08.430702 17029 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afbc6586a5334ab495863402267d84f7: Loaded TSK: 1
F20260928 05:15:08.431821 17029 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa70dac19ad google::LogMessage::Fail() at ??:0
@ 0x7fa70dac488a google::LogMessage::SendToLog() at ??:0
@ 0x7fa70dac14de google::LogMessage::Flush() at ??:0
@ 0x7fa70dac2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa7118257d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa71181473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa7234f01c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa7234ef783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa72357b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa72357b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa7235c55e9 std::function<>::operator()() at ??:0
@ 0x7fa7234f3cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa7234f1c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa723579610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa723579439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa71070d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa71071091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa710710799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa7106ebeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa7139c56db start_thread at ??:0
@ 0x7fa70c95b61f clone at ??:0
W20260928 05:15:08.750856 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:08.751158 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:08.751313 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:08.758896 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:08.759011 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:08.759210 17020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:08.768968 17020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37807
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:08.770314 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:08.771663 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:08.784598 17066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:08.784544 17065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:08.784565 17068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:08.785537 17020 server_base.cc:1034] running on GCE node
I20260928 05:15:08.786123 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:08.787153 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:08.788383 17020 hybrid_clock.cc:648] HybridClock initialized: now 1790572508788321 us; error 68 us; skew 500 ppm
I20260928 05:15:08.788700 17020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:08 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:08.791877 17020 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:15:08.793435 17020 webserver.cc:460] Webserver started at http://127.0.97.1:37807/ using document root <none> and password file <none>
I20260928 05:15:08.794129 17020 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:08.794276 17020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:08.799701 17020 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:15:08.802745 17075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:08.804101 17020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:15:08.804301 17020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "fa13c62f66e64997a67b4152872da138"
format_stamp: "Formatted at 2026-09-28 05:14:22 on dist-test-slave-6718"
server_key: "4b4ad92051923fb79ede7938893f9625"
server_key_iv: "d2ff00b8681868e9e0fe7131249beda0"
server_key_version: "encryptionkey@0"
I20260928 05:15:08.805264 17020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:08.840647 17020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:08.844954 17020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:08.845247 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:08.846269 17020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:08.848214 17082 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 05:15:08.854378 17020 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 05:15:08.854517 17020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 05:15:08.854709 17020 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 05:15:08.857949 17020 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 05:15:08.858058 17020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260928 05:15:08.858453 17082 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Bootstrap starting.
I20260928 05:15:08.868431 17082 log.cc:824] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:08.884984 17082 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 05:15:08.885984 17082 tablet_bootstrap.cc:492] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Bootstrap complete.
I20260928 05:15:08.886762 17082 ts_tablet_manager.cc:1397] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.015s
I20260928 05:15:08.897828 17082 raft_consensus.cc:348] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:15:08.899220 17082 raft_consensus.cc:729] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa13c62f66e64997a67b4152872da138, State: Initialized, Role: FOLLOWER
I20260928 05:15:08.900008 17082 consensus_queue.cc:260] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:15:08.901678 17082 ts_tablet_manager.cc:1428] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260928 05:15:08.903704 17020 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35479
I20260928 05:15:08.903770 17190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35479 every 8 connection(s)
I20260928 05:15:08.905086 17020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:08.910152 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:08.908078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:55079)
0928 05:15:08.908967 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:08.909157 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":434,"threads_started":1}
W20260928 05:15:08.910810 17191 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40623 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40623: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
I20260928 05:15:08.911625 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:08.911309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:32839)
0928 05:15:08.911455 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:08.911525 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:15:08.912437 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:08.912102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:52317)
0928 05:15:08.912263 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:08.912327 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:15:08.912669 17191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:15:08.915307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17020
I20260928 05:15:08.915916 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38445
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40623
--webserver_port=45397
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:09.244277 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:09.244562 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:09.244730 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:09.251936 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:09.252045 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:09.252239 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:09.261969 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45397
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:09.263335 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:09.264642 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:09.277762 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:09.277721 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:09.277793 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:09.278848 17194 server_base.cc:1034] running on GCE node
I20260928 05:15:09.279445 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:09.280373 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:09.281601 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790572509281556 us; error 62 us; skew 500 ppm
I20260928 05:15:09.281912 17194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:09 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:09.285136 17194 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:15:09.286752 17194 webserver.cc:460] Webserver started at http://127.0.97.2:45397/ using document root <none> and password file <none>
I20260928 05:15:09.287513 17194 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:09.287657 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:09.292999 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 05:15:09.295815 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:09.297147 17194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:09.297338 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "08be3a022fdf1a1dace982e4131f1726"
server_key_iv: "9d80bcc1567ae75a0467aeab21af37da"
server_key_version: "encryptionkey@0"
I20260928 05:15:09.298403 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:09.334154 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:09.338487 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:09.338814 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:09.339896 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:09.341279 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:09.341403 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:09.341542 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:09.341629 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:09.375800 17194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38445
I20260928 05:15:09.375893 17321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38445 every 8 connection(s)
I20260928 05:15:09.377275 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:09.379186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17194
I20260928 05:15:09.379884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35185
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40623
--webserver_port=44995
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:15:09.382939 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.380814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:36687)
0928 05:15:09.381676 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.381928 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":353,"threads_started":1}
W20260928 05:15:09.383587 17322 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40623 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40623: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
I20260928 05:15:09.384511 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.384190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:48709)
0928 05:15:09.384343 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.384410 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 05:15:09.385484 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.385078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:57381)
0928 05:15:09.385270 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.385351 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260928 05:15:09.386716 17322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 05:15:09.707298 17325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:09.707595 17325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:09.707755 17325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:09.715438 17325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:09.715555 17325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:09.715735 17325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:09.725639 17325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44995
--tserver_master_addrs=127.0.97.62:40623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:09.726970 17325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:09.728307 17325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:09.741516 17330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:09.741575 17331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:09.741684 17333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:09.742889 17325 server_base.cc:1034] running on GCE node
I20260928 05:15:09.743526 17325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:09.744495 17325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:09.745705 17325 hybrid_clock.cc:648] HybridClock initialized: now 1790572509745654 us; error 39 us; skew 500 ppm
I20260928 05:15:09.746033 17325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:09 dist-test-slave-6718 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:09.749274 17325 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:15:09.750792 17325 webserver.cc:460] Webserver started at http://127.0.97.3:44995/ using document root <none> and password file <none>
I20260928 05:15:09.751430 17325 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:09.751528 17325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:09.756721 17325 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 05:15:09.759480 17340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:09.760957 17325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 05:15:09.761132 17325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "d2c15b7a9e394e91b1defa2253f9a77f"
format_stamp: "Formatted at 2026-09-28 05:14:23 on dist-test-slave-6718"
server_key: "2e04e334ef59414f1c578c23e24c3d0e"
server_key_iv: "76342e89ab0916347efce16ba4c0d3fa"
server_key_version: "encryptionkey@0"
I20260928 05:15:09.762112 17325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:09.778131 17325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:09.782322 17325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:09.782642 17325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:09.783574 17325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:09.784865 17325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:09.784986 17325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:09.785099 17325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:09.785182 17325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:09.819375 17325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35185
I20260928 05:15:09.819459 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35185 every 8 connection(s)
I20260928 05:15:09.820842 17325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:09.823818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17325
I20260928 05:15:09.825847 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.823770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:52307)
0928 05:15:09.824614 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.824860 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":378,"threads_started":1}
W20260928 05:15:09.826500 17453 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40623 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40623: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
I20260928 05:15:09.827426 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.827080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:51401)
0928 05:15:09.827240 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.827321 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 05:15:09.828226 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.827900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:37197)
0928 05:15:09.828036 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.828113 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:15:09.828475 17453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 05:15:09.913991 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:09.913165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:40403)
0928 05:15:09.913647 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:09.913733 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":206,"threads_started":1}
I20260928 05:15:10.388084 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.387271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:58167)
0928 05:15:10.387752 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:10.387859 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":222,"threads_started":1}
I20260928 05:15:10.731663 17459 raft_consensus.cc:482] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:10.732005 17459 raft_consensus.cc:504] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:15:10.733757 17459 leader_election.cc:288] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185), 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
Sep 28 05:15:10 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 28 05:15:10 dist-test-slave-6718 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:15:10.757149 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.734055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35185 (local address 127.0.97.1:36925)
0928 05:15:10.735040 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:10.735077 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:10.735760 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:10.738474 (+ 2714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:10.738480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:10.739500 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:10.740261 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:10.740282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.742823 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.742828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:10.744312 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:10.744327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.745729 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.745736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:10.745945 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:10.746747 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:10.746780 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:10.749359 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:10.753331 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:10.753341 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:10.753358 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:10.753931 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:10.754416 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:10.754422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:10.754426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:10.754745 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:10.755544 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:10.755552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:10.755925 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 05:15:10.756240 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 05:15:10.756735 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":77,"spinlock_wait_cycles":24960,"thread_start_us":692,"threads_started":1}
I20260928 05:15:10.757450 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.735192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38445 (local address 127.0.97.1:43263)
0928 05:15:10.735803 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:10.735822 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:10.735941 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:10.738458 (+ 2517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:10.738471 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:10.739501 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:10.740262 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:10.740282 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.742847 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.742851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:10.744338 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:10.744351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.745658 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.745666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:10.745935 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:10.746836 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:10.746853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:10.749622 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:10.753557 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:10.753562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:10.753567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:10.753994 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:10.754442 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:10.754448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:10.754451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:10.754733 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:10.755544 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:10.755901 (+ 357us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:10.756023 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 05:15:10.756240 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 05:15:10.756936 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":118,"spinlock_wait_cycles":10496,"thread_start_us":286,"threads_started":1}
I20260928 05:15:10.757715 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.735518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43263 (local address 127.0.97.2:38445)
0928 05:15:10.736081 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:10.736096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:10.736138 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:15:10.737596 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:10.737604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:10.738022 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:10.738349 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:10.740474 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.742598 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.744516 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.745350 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.745571 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:10.749896 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:10.749903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:10.749916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:10.749952 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:10.753324 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:10.754134 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:10.754150 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:10.754158 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:10.754312 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:10.754849 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:10.754855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:10.754858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:10.755404 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:10.755611 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:10.756499 (+ 888us) server_negotiation.cc:299] Negotiation successful
0928 05:15:10.756944 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":230,"threads_started":1}
I20260928 05:15:10.758483 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.734349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36925 (local address 127.0.97.3:35185)
0928 05:15:10.735065 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:10.735086 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:10.735213 (+ 127us) server_negotiation.cc:407] Connection header received
0928 05:15:10.737806 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:10.737814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:10.738185 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:10.738564 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:10.740537 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.742589 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.744475 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:10.745473 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:10.745742 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:10.749692 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:10.749701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:10.749716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:10.749756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:10.753146 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:10.754099 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:10.754106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:10.754112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:10.754288 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:10.754876 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:10.754887 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:10.754889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:10.755397 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:10.755611 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:10.757700 (+ 2089us) server_negotiation.cc:299] Negotiation successful
0928 05:15:10.758167 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":370,"threads_started":1}
W20260928 05:15:10.761348 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
W20260928 05:15:10.761693 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:15:10.761829 17076 leader_election.cc:302] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa13c62f66e64997a67b4152872da138; no voters: 9fcfe1bb5f3b4e0eae15c8191dde0dac, d2c15b7a9e394e91b1defa2253f9a77f
I20260928 05:15:10.762257 17459 raft_consensus.cc:2726] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:15:10.829900 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.829025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:46651)
0928 05:15:10.829514 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:10.829635 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
I20260928 05:15:10.915459 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:10.915057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:42251)
0928 05:15:10.915226 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:10.915316 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 05:15:11.389744 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:11.389054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:41923)
0928 05:15:11.389485 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:11.389563 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260928 05:15:11.831707 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:11.830937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:52933)
0928 05:15:11.831477 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:11.831550 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":299,"threads_started":1}
I20260928 05:15:11.917057 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:11.916353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:60497)
0928 05:15:11.916803 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:11.916883 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260928 05:15:12.391326 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:12.390638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:49289)
0928 05:15:12.391037 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:12.391120 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":216,"threads_started":1}
I20260928 05:15:12.621367 17469 raft_consensus.cc:482] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:12.621592 17469 raft_consensus.cc:504] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:15:12.622290 17469 leader_election.cc:288] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185), 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
W20260928 05:15:12.623421 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
W20260928 05:15:12.623624 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:15:12.623699 17076 leader_election.cc:302] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa13c62f66e64997a67b4152872da138; no voters: 9fcfe1bb5f3b4e0eae15c8191dde0dac, d2c15b7a9e394e91b1defa2253f9a77f
I20260928 05:15:12.623899 17469 raft_consensus.cc:2726] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:15:12.833182 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:12.832504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:43335)
0928 05:15:12.832945 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:12.833024 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":233,"threads_started":1}
I20260928 05:15:12.918673 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:12.917924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:33653)
0928 05:15:12.918366 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:12.918446 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260928 05:15:13.392938 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:13.392217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:51091)
0928 05:15:13.392686 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:13.392772 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":214,"threads_started":1}
I20260928 05:15:13.834898 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:13.834112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:54907)
0928 05:15:13.834636 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:13.834715 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260928 05:15:13.920390 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:13.919623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:43523)
0928 05:15:13.920101 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:13.920205 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":216,"threads_started":1}
I20260928 05:15:14.394732 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:14.393891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:50853)
0928 05:15:14.394398 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:14.394479 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
I20260928 05:15:14.836529 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:14.835808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:38965)
0928 05:15:14.836289 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:14.836370 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":271,"threads_started":1}
I20260928 05:15:14.875952 17477 raft_consensus.cc:482] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:14.876183 17477 raft_consensus.cc:504] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:15:14.876880 17477 leader_election.cc:288] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185), 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
W20260928 05:15:14.877869 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
W20260928 05:15:14.878104 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:15:14.878281 17076 leader_election.cc:302] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa13c62f66e64997a67b4152872da138; no voters: 9fcfe1bb5f3b4e0eae15c8191dde0dac, d2c15b7a9e394e91b1defa2253f9a77f
I20260928 05:15:14.878551 17477 raft_consensus.cc:2726] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:15:14.922400 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:14.921488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:52577)
0928 05:15:14.922069 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:14.922177 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":334,"threads_started":1}
I20260928 05:15:15.396349 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:15.395573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:39835)
0928 05:15:15.396057 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:15.396137 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
I20260928 05:15:15.838207 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:15.837435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:59943)
0928 05:15:15.837936 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:15.838016 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
I20260928 05:15:15.924041 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:15.923309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:48057)
0928 05:15:15.923754 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:15.923843 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":218,"threads_started":1}
I20260928 05:15:16.398049 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:16.397239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:33817)
0928 05:15:16.397749 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:16.397853 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":232,"threads_started":1}
I20260928 05:15:16.840368 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:16.839394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:57433)
0928 05:15:16.840074 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:16.840181 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":341,"threads_started":1}
I20260928 05:15:16.925709 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:16.924972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:34207)
0928 05:15:16.925460 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:16.925552 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260928 05:15:17.400038 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:17.399147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:39783)
0928 05:15:17.399740 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:17.399853 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":315,"threads_started":1}
I20260928 05:15:17.842147 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:17.841412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:45801)
0928 05:15:17.841875 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:17.841960 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260928 05:15:17.927469 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:17.926753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:58473)
0928 05:15:17.927201 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:17.927282 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260928 05:15:18.401643 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:18.400917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:57567)
0928 05:15:18.401393 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:18.401476 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260928 05:15:18.843827 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:18.843124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:53791)
0928 05:15:18.843588 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:18.843672 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260928 05:15:18.929220 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:18.928484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:55105)
0928 05:15:18.928964 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:18.929052 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
I20260928 05:15:19.403375 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:19.402670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:37195)
0928 05:15:19.403125 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:19.403204 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260928 05:15:19.845629 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:19.844771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:43603)
0928 05:15:19.845257 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:19.845354 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
I20260928 05:15:19.893811 17493 raft_consensus.cc:482] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:19.894053 17493 raft_consensus.cc:504] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c15b7a9e394e91b1defa2253f9a77f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35185 } } peers { permanent_uuid: "fa13c62f66e64997a67b4152872da138" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35479 } } peers { permanent_uuid: "9fcfe1bb5f3b4e0eae15c8191dde0dac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38445 } }
I20260928 05:15:19.894798 17493 leader_election.cc:288] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185), 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445)
W20260928 05:15:19.895746 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2c15b7a9e394e91b1defa2253f9a77f (127.0.97.3:35185): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
W20260928 05:15:19.895974 17076 leader_election.cc:341] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9fcfe1bb5f3b4e0eae15c8191dde0dac (127.0.97.2:38445): Not found: Tablet not found: 3d458948ba7d4895aac49c043ecdfb3a
I20260928 05:15:19.896054 17076 leader_election.cc:302] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa13c62f66e64997a67b4152872da138; no voters: 9fcfe1bb5f3b4e0eae15c8191dde0dac, d2c15b7a9e394e91b1defa2253f9a77f
I20260928 05:15:19.896324 17493 raft_consensus.cc:2726] T 3d458948ba7d4895aac49c043ecdfb3a P fa13c62f66e64997a67b4152872da138 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 05:15:19.931082 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:19.930177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:53893)
0928 05:15:19.930770 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:19.930852 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260928 05:15:20.405243 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:20.404345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:42861)
0928 05:15:20.404910 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:20.405015 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":315,"threads_started":1}
I20260928 05:15:20.847419 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:20.846686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:33509)
0928 05:15:20.847153 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:20.847236 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260928 05:15:20.932710 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:20.931991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:34071)
0928 05:15:20.932458 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:20.932546 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260928 05:15:21.406900 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:21.406168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:52693)
0928 05:15:21.406646 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:21.406727 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260928 05:15:21.849305 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:21.848556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:35003)
0928 05:15:21.849007 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:21.849090 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260928 05:15:21.934475 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:21.933760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:35699)
0928 05:15:21.934205 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:21.934295 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260928 05:15:22.408766 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:22.407935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:48897)
0928 05:15:22.408503 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:22.408589 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":325,"threads_started":1}
I20260928 05:15:22.850957 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:22.850218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:42083)
0928 05:15:22.850706 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:22.850796 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":226,"threads_started":1}
I20260928 05:15:22.936210 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:22.935504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:46901)
0928 05:15:22.935944 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:22.936027 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":236,"threads_started":1}
I20260928 05:15:23.410640 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:23.409844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:45887)
0928 05:15:23.410294 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:23.410375 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260928 05:15:23.852499 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:23.851813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:40483)
0928 05:15:23.852226 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:23.852315 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":220,"threads_started":1}
I20260928 05:15:23.938354 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:23.937519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:55389)
0928 05:15:23.938023 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:23.938120 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
I20260928 05:15:24.412473 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:24.411798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.2:36207)
0928 05:15:24.412222 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:24.412314 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":203,"threads_started":1}
I20260928 05:15:24.830150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17020
I20260928 05:15:24.849936 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:24.831414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.0.97.1:35479)
0928 05:15:24.832080 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:24.832091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:24.832125 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:15:24.832216 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:24.832235 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:24.832547 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:24.832729 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:24.834167 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:24.835295 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:24.836638 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:24.837240 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:24.837359 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:24.843660 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:24.843675 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:24.843690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:24.843732 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:24.847077 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:24.847861 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:24.847869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:24.847871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:24.848017 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:24.848341 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:24.848344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:24.848346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:24.848685 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:24.848911 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:24.849306 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 05:15:24.849610 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":326,"threads_started":1}
I20260928 05:15:24.851358 17185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:24.865672 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:24.853730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.3:36473)
0928 05:15:24.865343 (+ 11613us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:24.865464 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11473,"thread_start_us":11333,"threads_started":1}
I20260928 05:15:24.941792 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:24.940653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40623 (local address 127.0.97.1:56493)
0928 05:15:24.941294 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:24.941426 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":291,"threads_started":1}
I20260928 05:15:24.941843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17020
I20260928 05:15:24.960047 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17194
I20260928 05:15:24.973991 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:24.961030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.97.2:38445)
0928 05:15:24.961562 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:24.961567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:24.961583 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:15:24.961636 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:24.961639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:24.961696 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:24.961794 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:24.963253 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:24.964063 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:24.965360 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:24.965787 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:24.965880 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:24.968717 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:24.968722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:24.968726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:24.968759 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:24.971552 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:24.972210 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:24.972214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:24.972216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:24.972375 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:24.972694 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:24.972697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:24.972699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:24.972972 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:24.973202 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:24.973419 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 05:15:24.973580 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":301,"threads_started":1}
I20260928 05:15:24.975019 17317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:25.058976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17194
I20260928 05:15:25.076308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17325
I20260928 05:15:25.089699 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:25.077072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.0.97.3:35185)
0928 05:15:25.077603 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:25.077608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:25.077624 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:15:25.077678 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:25.077682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:25.077760 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:25.077853 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:25.079221 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:25.079998 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:25.081245 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:25.081689 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:25.081798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:25.084524 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:25.084530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:25.084534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:25.084566 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:25.087412 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:25.088015 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:25.088019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:25.088021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:25.088145 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:25.088447 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:25.088451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:25.088452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:25.088687 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:25.088873 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:25.089107 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 05:15:25.089288 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
I20260928 05:15:25.090390 17448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:25.173022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17325
2026-09-28T05:15:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64539 ms)
[----------] 30 tests from SecurityITest (296456 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:25 dist-test-slave-6718 krb5kdc[17518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:25 dist-test-slave-6718 krb5kdc[17518](info): set up 2 sockets
Sep 28 05:15:25 dist-test-slave-6718 krb5kdc[17518](info): commencing operation
krb5kdc: starting...
W20260928 05:15:26.278733 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:26 dist-test-slave-6718 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:15:26.532465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42375
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:26.863677 17531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:26.864053 17531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:26.864213 17531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:26.871605 17531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:26.871731 17531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:26.871778 17531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:26.871829 17531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:26.871872 17531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:26.881963 17531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:26.883277 17531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:26.885110 17531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:26.898278 17537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:26.898264 17539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:26.898319 17536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:26.899163 17531 server_base.cc:1034] running on GCE node
I20260928 05:15:26.900012 17531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:26.901556 17531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:26.902787 17531 hybrid_clock.cc:648] HybridClock initialized: now 1790572526902728 us; error 52 us; skew 500 ppm
I20260928 05:15:26.903147 17531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:26 dist-test-slave-6718 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:26.906279 17531 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:15:26.907935 17531 webserver.cc:460] Webserver started at http://127.0.97.62:37687/ using document root <none> and password file <none>
I20260928 05:15:26.908658 17531 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:26.908804 17531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:26.909121 17531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:26.912155 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "6d93f2a735d740eb892e0387dfbf161f"
format_stamp: "Formatted at 2026-09-28 05:15:26 on dist-test-slave-6718"
server_key: "33a0698cb527413ee77359dfa06cb261"
server_key_iv: "612497ce20d092c9bf28b43f9bdb8374"
server_key_version: "encryptionkey@0"
I20260928 05:15:26.912920 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d93f2a735d740eb892e0387dfbf161f"
format_stamp: "Formatted at 2026-09-28 05:15:26 on dist-test-slave-6718"
server_key: "33a0698cb527413ee77359dfa06cb261"
server_key_iv: "612497ce20d092c9bf28b43f9bdb8374"
server_key_version: "encryptionkey@0"
I20260928 05:15:26.918182 17531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:15:26.922000 17546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:26.923568 17531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:26.923770 17531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "6d93f2a735d740eb892e0387dfbf161f"
format_stamp: "Formatted at 2026-09-28 05:15:26 on dist-test-slave-6718"
server_key: "33a0698cb527413ee77359dfa06cb261"
server_key_iv: "612497ce20d092c9bf28b43f9bdb8374"
server_key_version: "encryptionkey@0"
I20260928 05:15:26.923988 17531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:26.945137 17531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:26.949869 17531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:26.950174 17531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:26.969413 17531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42375
I20260928 05:15:26.969465 17597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42375 every 8 connection(s)
I20260928 05:15:26.970894 17531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:26.974748 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:26.976583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17531
I20260928 05:15:26.976795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:26.977481 388 external_mini_cluster.cc:1428] Setting key 198a43a69f0d6b14cd5973f58a46984b
I20260928 05:15:26.987006 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f: Bootstrap starting.
I20260928 05:15:26.990923 17598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:26.992296 17598 log.cc:824] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:15:26 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:26.995533 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f: No bootstrap required, opened a new log
I20260928 05:15:27.001897 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:26.979555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.0.97.62:42375)
0928 05:15:26.980437 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:26.980450 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:26.980497 (+ 47us) server_negotiation.cc:407] Connection header received
0928 05:15:26.982189 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:26.982212 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:26.982712 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:26.983067 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:26.984919 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:26.986315 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:26.988179 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:26.988925 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:26.989066 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:26.994307 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:26.994315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:26.994330 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:26.994370 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:26.997643 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:26.998521 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:26.998571 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:26.998581 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:26.998758 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:26.999158 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:26.999161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:26.999163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:26.999684 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:26.999896 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:27.000415 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 05:15:27.000857 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":404,"threads_started":1}
I20260928 05:15:27.004959 17598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d93f2a735d740eb892e0387dfbf161f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42375 } }
I20260928 05:15:27.005334 17598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:27.005437 17598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d93f2a735d740eb892e0387dfbf161f, State: Initialized, Role: FOLLOWER
I20260928 05:15:27.005965 17598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d93f2a735d740eb892e0387dfbf161f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42375 } }
I20260928 05:15:27.006217 17598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:27.006359 17598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:27.006558 17598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:27.008811 17598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d93f2a735d740eb892e0387dfbf161f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42375 } }
I20260928 05:15:27.009354 17598 leader_election.cc:302] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d93f2a735d740eb892e0387dfbf161f; no voters:
I20260928 05:15:27.009927 17598 leader_election.cc:288] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:27.010094 17603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:27.010859 17603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [term 1 LEADER]: Becoming Leader. State: Replica: 6d93f2a735d740eb892e0387dfbf161f, State: Running, Role: LEADER
I20260928 05:15:27.011487 17603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d93f2a735d740eb892e0387dfbf161f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42375 } }
I20260928 05:15:27.012004 17598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:27.014428 17605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d93f2a735d740eb892e0387dfbf161f. Latest consensus state: current_term: 1 leader_uuid: "6d93f2a735d740eb892e0387dfbf161f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d93f2a735d740eb892e0387dfbf161f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42375 } } }
I20260928 05:15:27.014945 17605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:27.014793 17604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d93f2a735d740eb892e0387dfbf161f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d93f2a735d740eb892e0387dfbf161f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42375 } } }
I20260928 05:15:27.015125 17604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:27.015916 17610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:27.021659 17610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:27.035629 17610 catalog_manager.cc:1347] Generated new cluster ID: cab9830bc6ef44828af27ea935cd0402
I20260928 05:15:27.035745 17610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:27.066681 17610 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:27.068849 17610 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:27.081503 17610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f: Generated new TSK 0
I20260928 05:15:27.082674 17610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:15:27.146209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42375
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:27.484023 17624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:27.484295 17624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:27.484468 17624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:27.491760 17624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:27.491873 17624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:27.492101 17624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:27.503880 17624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:27.505199 17624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:27.506570 17624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:27.520057 17629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:27.520095 17632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:27.520057 17630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:27.520671 17624 server_base.cc:1034] running on GCE node
I20260928 05:15:27.521332 17624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:27.522301 17624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:27.523535 17624 hybrid_clock.cc:648] HybridClock initialized: now 1790572527523451 us; error 78 us; skew 500 ppm
I20260928 05:15:27.523876 17624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:27 dist-test-slave-6718 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:27.527289 17624 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:15:27.528975 17624 webserver.cc:460] Webserver started at http://127.0.97.1:41977/ using document root <none> and password file <none>
I20260928 05:15:27.529618 17624 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:27.529711 17624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:27.529963 17624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:27.532806 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "8140d39711ab4f27b6eb326a3b69db5d"
format_stamp: "Formatted at 2026-09-28 05:15:27 on dist-test-slave-6718"
server_key: "3306128a6c4c9c60c7a460912f7f7daf"
server_key_iv: "d9cf0f33e5dd34aafdd3582a017f2bc5"
server_key_version: "encryptionkey@0"
I20260928 05:15:27.533494 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8140d39711ab4f27b6eb326a3b69db5d"
format_stamp: "Formatted at 2026-09-28 05:15:27 on dist-test-slave-6718"
server_key: "3306128a6c4c9c60c7a460912f7f7daf"
server_key_iv: "d9cf0f33e5dd34aafdd3582a017f2bc5"
server_key_version: "encryptionkey@0"
I20260928 05:15:27.538374 17624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 05:15:27.541721 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:27.543061 17624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:27.543226 17624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "8140d39711ab4f27b6eb326a3b69db5d"
format_stamp: "Formatted at 2026-09-28 05:15:27 on dist-test-slave-6718"
server_key: "3306128a6c4c9c60c7a460912f7f7daf"
server_key_iv: "d9cf0f33e5dd34aafdd3582a017f2bc5"
server_key_version: "encryptionkey@0"
I20260928 05:15:27.543433 17624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:27.557008 17624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:27.561007 17624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:27.561316 17624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:27.562227 17624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:27.563584 17624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:27.563692 17624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:27.563840 17624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:27.563917 17624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:27.597237 17624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45667
I20260928 05:15:27.597297 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45667 every 8 connection(s)
I20260928 05:15:27.598606 17624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:27.601171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17624
I20260928 05:15:27.601397 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:15:27.602217 388 external_mini_cluster.cc:1428] Setting key 192c38a04666b64aed8e4abb05555785
Sep 28 05:15:27 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:27.620501 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:27.601364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55259 (local address 127.0.97.62:42375)
0928 05:15:27.604315 (+ 2951us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:27.604322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:27.604344 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:15:27.605692 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:27.605697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:27.605794 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:27.605914 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:27.607526 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:27.608255 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:27.609895 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:27.610311 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:27.610454 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:27.614487 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:27.614492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:27.614495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:27.614574 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:27.617297 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:27.618053 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:27.618058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:27.618060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:27.618211 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:27.618706 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:27.618717 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:27.618720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:27.618982 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:27.619139 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:27.619866 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 05:15:27.620096 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2817,"thread_start_us":2703,"threads_started":1}
I20260928 05:15:27.621443 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:27.601969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42375 (local address 127.0.97.1:55259)
0928 05:15:27.603542 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:27.603589 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:27.605420 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:27.606154 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:27.606164 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:27.606770 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:27.607357 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:27.607375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:27.608436 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:27.608440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:27.609726 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:27.609739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:27.610583 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:27.610591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:27.610833 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:27.611558 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:27.611580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:27.614246 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:27.617452 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:27.617463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:27.617483 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:27.617913 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:27.618332 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:27.618338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:27.618341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:27.618587 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:27.619111 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:27.619119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:27.619487 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 05:15:27.619747 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 05:15:27.620312 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":1065,"threads_started":1}
I20260928 05:15:27.623131 17752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42375
I20260928 05:15:27.623469 17752 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:27.624181 17752 heartbeater.cc:511] Master 127.0.97.62:42375 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:15:27.626323 17563 ts_manager.cc:194] Registered new tserver with Master: 8140d39711ab4f27b6eb326a3b69db5d (127.0.97.1:45667)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:15:27.628115 17563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:15:27.657245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42375
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:27.981783 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:27.982115 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:27.982282 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:27.989663 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:27.989813 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:27.990003 17758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:27.999959 17758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:28.001263 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:28.002573 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:28.015497 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:28.015590 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:28.015614 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:28.016615 17758 server_base.cc:1034] running on GCE node
I20260928 05:15:28.017215 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:28.018215 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:28.019439 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790572528019371 us; error 59 us; skew 500 ppm
I20260928 05:15:28.019762 17758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:28 dist-test-slave-6718 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:28.022922 17758 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:15:28.024415 17758 webserver.cc:460] Webserver started at http://127.0.97.2:41685/ using document root <none> and password file <none>
I20260928 05:15:28.025125 17758 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:28.025266 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:28.025586 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:28.028697 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "c29ba633ac0a485998336c98e62592f9"
format_stamp: "Formatted at 2026-09-28 05:15:28 on dist-test-slave-6718"
server_key: "cc0bbcf95926216f23ac54a4524290bf"
server_key_iv: "f2c67ba38e29d4b276014ccb4e5c7bc2"
server_key_version: "encryptionkey@0"
I20260928 05:15:28.029536 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c29ba633ac0a485998336c98e62592f9"
format_stamp: "Formatted at 2026-09-28 05:15:28 on dist-test-slave-6718"
server_key: "cc0bbcf95926216f23ac54a4524290bf"
server_key_iv: "f2c67ba38e29d4b276014ccb4e5c7bc2"
server_key_version: "encryptionkey@0"
I20260928 05:15:28.035101 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 05:15:28.038579 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:28.040042 17758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:28.040244 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "c29ba633ac0a485998336c98e62592f9"
format_stamp: "Formatted at 2026-09-28 05:15:28 on dist-test-slave-6718"
server_key: "cc0bbcf95926216f23ac54a4524290bf"
server_key_iv: "f2c67ba38e29d4b276014ccb4e5c7bc2"
server_key_version: "encryptionkey@0"
I20260928 05:15:28.040468 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:28.062294 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:28.066388 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:28.066725 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:28.067700 17758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:28.069010 17758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:28.069121 17758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:28.069268 17758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:28.069340 17758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:28.103117 17758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35625
I20260928 05:15:28.103207 17885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35625 every 8 connection(s)
I20260928 05:15:28.104411 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:28.111116 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17758
I20260928 05:15:28.111294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:15:28.111840 388 external_mini_cluster.cc:1428] Setting key e62196d3730c0b4509867e8e7868ba95
Sep 28 05:15:28 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:28.131034 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.107326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60477 (local address 127.0.97.62:42375)
0928 05:15:28.107529 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.107535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.110215 (+ 2680us) server_negotiation.cc:407] Connection header received
0928 05:15:28.112895 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.112901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.113001 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.113110 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:28.115097 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.115785 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.118248 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.118887 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.118994 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.123469 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:28.123478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:28.123486 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:28.123534 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:28.127326 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.128151 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.128159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.128165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.128331 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.128907 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.128915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.128921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.129294 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:28.129441 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.130265 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.130568 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 05:15:28.131861 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.107864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42375 (local address 127.0.97.2:60477)
0928 05:15:28.110039 (+ 2175us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.110084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.112542 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.113385 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.113396 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.114136 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:28.114901 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.114927 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.115947 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.115955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.118104 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.118122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.119102 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.119111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.119434 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.120147 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:28.120186 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:28.123210 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:28.127517 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:28.127524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:28.127541 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:28.128003 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:28.128476 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:28.128480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:28.128483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:28.128779 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:28.129429 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:28.129435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:28.129829 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.130104 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.130616 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1880,"thread_start_us":1678,"threads_started":1}
I20260928 05:15:28.133488 17886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42375
I20260928 05:15:28.133841 17886 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:28.134707 17886 heartbeater.cc:511] Master 127.0.97.62:42375 requested a full tablet report, sending...
I20260928 05:15:28.136348 17563 ts_manager.cc:194] Registered new tserver with Master: c29ba633ac0a485998336c98e62592f9 (127.0.97.2:35625)
I20260928 05:15:28.137192 17563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60477
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:15:28.170415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42375
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:28.506098 17891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:28.506448 17891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:28.506656 17891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:28.514051 17891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:28.514209 17891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:28.514391 17891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:28.524147 17891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:28.525494 17891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:28.526888 17891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:28.540503 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:28.540598 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:28.540531 17897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:28.544090 17891 server_base.cc:1034] running on GCE node
I20260928 05:15:28.544956 17891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:28.546125 17891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:28.547387 17891 hybrid_clock.cc:648] HybridClock initialized: now 1790572528547346 us; error 55 us; skew 500 ppm
I20260928 05:15:28.547813 17891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:28 dist-test-slave-6718 krb5kdc[17518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:28.551350 17891 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:15:28.552939 17891 webserver.cc:460] Webserver started at http://127.0.97.3:39741/ using document root <none> and password file <none>
I20260928 05:15:28.553642 17891 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:28.553740 17891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:28.553996 17891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:28.557008 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "760599a4c7da4cbdacf34ffbb854943d"
format_stamp: "Formatted at 2026-09-28 05:15:28 on dist-test-slave-6718"
server_key: "bd9da9769878d1e032352c64a076624c"
server_key_iv: "95dc1ac04b155d2931e8d01ec539f696"
server_key_version: "encryptionkey@0"
I20260928 05:15:28.557770 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "760599a4c7da4cbdacf34ffbb854943d"
format_stamp: "Formatted at 2026-09-28 05:15:28 on dist-test-slave-6718"
server_key: "bd9da9769878d1e032352c64a076624c"
server_key_iv: "95dc1ac04b155d2931e8d01ec539f696"
server_key_version: "encryptionkey@0"
I20260928 05:15:28.563133 17891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:15:28.566771 17906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:28.568269 17891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:28.568437 17891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "760599a4c7da4cbdacf34ffbb854943d"
format_stamp: "Formatted at 2026-09-28 05:15:28 on dist-test-slave-6718"
server_key: "bd9da9769878d1e032352c64a076624c"
server_key_iv: "95dc1ac04b155d2931e8d01ec539f696"
server_key_version: "encryptionkey@0"
I20260928 05:15:28.568610 17891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:28.598682 17891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:28.603274 17891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:28.603596 17891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:28.604598 17891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:28.606142 17891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:28.606247 17891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:28.606405 17891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:28.606490 17891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:28.631094 17752 heartbeater.cc:503] Master 127.0.97.62:42375 was elected leader, sending a full tablet report...
I20260928 05:15:28.641507 17891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44657
I20260928 05:15:28.641594 18018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44657 every 8 connection(s)
I20260928 05:15:28.642865 17891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:28.644522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 17891
I20260928 05:15:28.644696 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:15:28.645246 388 external_mini_cluster.cc:1428] Setting key 97b7835cb252fbca181f064e8a5c4866
Sep 28 05:15:28 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:28.663530 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.645750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55945 (local address 127.0.97.62:42375)
0928 05:15:28.646249 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.646256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.647522 (+ 1266us) server_negotiation.cc:407] Connection header received
0928 05:15:28.649414 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.649417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.649481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.649569 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:28.651032 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.651791 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.653437 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.653820 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.653912 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.657370 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:28.657375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:28.657378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:28.657413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:28.660619 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.661427 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.661431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.661433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.661549 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.661952 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.661955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.661957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.662213 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:28.662362 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.663071 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.663229 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260928 05:15:28.664505 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.646367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42375 (local address 127.0.97.3:55945)
0928 05:15:28.647339 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.647412 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.649169 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.649736 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.649743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.650246 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:28.650886 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.650904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.651964 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.651968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.653270 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.653282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.654007 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.654012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.654205 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.654865 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:28.654887 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:28.657132 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:28.660793 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:28.660800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:28.660813 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:28.661311 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:28.661643 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:28.661647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:28.661649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:28.661853 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:28.662327 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:28.662332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:28.662733 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.662961 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.663350 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":436,"threads_started":1}
I20260928 05:15:28.665825 18019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42375
I20260928 05:15:28.666129 18019 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:28.666918 18019 heartbeater.cc:511] Master 127.0.97.62:42375 requested a full tablet report, sending...
I20260928 05:15:28.668315 17563 ts_manager.cc:194] Registered new tserver with Master: 760599a4c7da4cbdacf34ffbb854943d (127.0.97.3:44657)
I20260928 05:15:28.669116 17563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55945
I20260928 05:15:28.673269 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:15:28.691164 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.677878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.0.97.62:42375)
0928 05:15:28.678109 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.678116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.678401 (+ 285us) server_negotiation.cc:407] Connection header received
0928 05:15:28.678614 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.678619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.678709 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.678814 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:28.680288 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.681444 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.682836 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.683254 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.683396 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.685830 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:28.685837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:28.685841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:28.685885 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:28.688401 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.689099 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.689106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.689110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.689237 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.689657 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.689663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.689666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.690001 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:28.690275 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.690611 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.690844 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:15:28.695251 17563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:15:28.702476 17563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:15:28.726501 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.715912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.0.97.1:45667)
0928 05:15:28.716609 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.716618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.717174 (+ 556us) server_negotiation.cc:407] Connection header received
0928 05:15:28.718387 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.718397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.718707 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.718908 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:28.719818 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.721222 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.723818 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.724859 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.725004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.725262 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.725919 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.726120 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":326,"threads_started":1}
I20260928 05:15:28.727597 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.716608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.0.97.3:44657)
0928 05:15:28.717294 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.717305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.718991 (+ 1686us) server_negotiation.cc:407] Connection header received
0928 05:15:28.719281 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.719291 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.719512 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.719639 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:28.720712 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.722248 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.724980 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.725978 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.726102 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.726325 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.727084 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.727250 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":307,"threads_started":1}
I20260928 05:15:28.727617 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.716060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44657 (local address 127.0.0.1:33058)
0928 05:15:28.718871 (+ 2811us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.718889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.719095 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.720014 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.720018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.720045 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:28.720510 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.720522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.722444 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.722449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.724826 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.724839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.726459 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.726466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.726887 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.726914 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.727029 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.727169 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"mutex_wait_us":27,"thread_start_us":2586,"threads_started":1}
I20260928 05:15:28.727919 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.715509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45667 (local address 127.0.0.1:33576)
0928 05:15:28.716997 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.717073 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.717341 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.719207 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.719217 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.719250 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:28.719602 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.719618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.721356 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.721364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.723655 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.723670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.725395 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.725402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.725628 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.725688 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.725840 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.726058 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":123,"thread_start_us":1137,"threads_started":1}
I20260928 05:15:28.728296 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.717719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.0.97.2:35625)
0928 05:15:28.718320 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.718328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.719037 (+ 709us) server_negotiation.cc:407] Connection header received
0928 05:15:28.719330 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.719340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.719582 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.719754 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:28.720454 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.722003 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.724856 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.726318 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.726460 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.726799 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.727341 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.727534 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":370,"threads_started":1}
I20260928 05:15:28.729022 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.717454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35625 (local address 127.0.0.1:33250)
0928 05:15:28.718939 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.718956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.719140 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.719840 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.719847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.719870 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:28.720278 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.720287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.722173 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.722178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.724661 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.724683 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.726950 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.726957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.727122 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.727154 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.728505 (+ 1351us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.728655 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":96,"thread_start_us":278,"threads_started":1}
I20260928 05:15:28.731699 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 07a507cc7f1a44c0b8a789cffaaf7321 (DEFAULT_TABLE table=test-table [id=81d97ea83bb14ee2838319a0934b895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:28.731696 17954 tablet_service.cc:1467] Processing CreateTablet for tablet 07a507cc7f1a44c0b8a789cffaaf7321 (DEFAULT_TABLE table=test-table [id=81d97ea83bb14ee2838319a0934b895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:28.733122 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a507cc7f1a44c0b8a789cffaaf7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:28.733178 17954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a507cc7f1a44c0b8a789cffaaf7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:28.734230 17821 tablet_service.cc:1467] Processing CreateTablet for tablet 07a507cc7f1a44c0b8a789cffaaf7321 (DEFAULT_TABLE table=test-table [id=81d97ea83bb14ee2838319a0934b895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:28.735666 17821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a507cc7f1a44c0b8a789cffaaf7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:28.746600 18038 tablet_bootstrap.cc:492] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Bootstrap starting.
I20260928 05:15:28.747663 18039 tablet_bootstrap.cc:492] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Bootstrap starting.
I20260928 05:15:28.749600 18040 tablet_bootstrap.cc:492] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Bootstrap starting.
I20260928 05:15:28.750439 18038 tablet_bootstrap.cc:654] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:28.751434 18039 tablet_bootstrap.cc:654] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:28.751801 18038 log.cc:824] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:28.752476 18040 tablet_bootstrap.cc:654] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:28.752737 18039 log.cc:824] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:28.753635 18040 log.cc:824] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:28.755620 18038 tablet_bootstrap.cc:492] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: No bootstrap required, opened a new log
I20260928 05:15:28.755941 18038 ts_tablet_manager.cc:1397] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 05:15:28.757272 18040 tablet_bootstrap.cc:492] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: No bootstrap required, opened a new log
I20260928 05:15:28.757555 18040 ts_tablet_manager.cc:1397] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 05:15:28.757644 18039 tablet_bootstrap.cc:492] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: No bootstrap required, opened a new log
I20260928 05:15:28.757913 18039 ts_tablet_manager.cc:1397] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 05:15:28.764575 18040 raft_consensus.cc:348] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.764995 18040 raft_consensus.cc:374] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:28.765098 18040 raft_consensus.cc:729] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29ba633ac0a485998336c98e62592f9, State: Initialized, Role: FOLLOWER
I20260928 05:15:28.765781 18040 consensus_queue.cc:260] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.766988 18038 raft_consensus.cc:348] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.767453 18038 raft_consensus.cc:374] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:28.767557 18038 raft_consensus.cc:729] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8140d39711ab4f27b6eb326a3b69db5d, State: Initialized, Role: FOLLOWER
I20260928 05:15:28.768365 17886 heartbeater.cc:503] Master 127.0.97.62:42375 was elected leader, sending a full tablet report...
I20260928 05:15:28.768283 18038 consensus_queue.cc:260] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.768566 18039 raft_consensus.cc:348] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.769062 18039 raft_consensus.cc:374] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:28.769109 18040 ts_tablet_manager.cc:1428] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 05:15:28.769172 18039 raft_consensus.cc:729] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760599a4c7da4cbdacf34ffbb854943d, State: Initialized, Role: FOLLOWER
I20260928 05:15:28.769913 18039 consensus_queue.cc:260] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.771247 18019 heartbeater.cc:503] Master 127.0.97.62:42375 was elected leader, sending a full tablet report...
I20260928 05:15:28.771925 18039 ts_tablet_manager.cc:1428] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260928 05:15:28.773893 18038 ts_tablet_manager.cc:1428] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20260928 05:15:28.810588 18045 raft_consensus.cc:482] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:28.810963 18045 raft_consensus.cc:504] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.813028 18045 leader_election.cc:288] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c29ba633ac0a485998336c98e62592f9 (127.0.97.2:35625), 760599a4c7da4cbdacf34ffbb854943d (127.0.97.3:44657)
I20260928 05:15:28.821127 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.813652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54055 (local address 127.0.97.2:35625)
0928 05:15:28.814066 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.814074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.814687 (+ 613us) server_negotiation.cc:407] Connection header received
0928 05:15:28.814924 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.814933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.815018 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.815137 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:28.816004 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.816791 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.819186 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.820033 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.820205 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.820342 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.820679 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.820816 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260928 05:15:28.821132 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.813626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33403 (local address 127.0.97.3:44657)
0928 05:15:28.813881 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.813890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.814695 (+ 805us) server_negotiation.cc:407] Connection header received
0928 05:15:28.815020 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.815024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.815112 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.815209 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:28.816049 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.816833 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.819178 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.820042 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.820203 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.820348 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.820705 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.820848 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 05:15:28.821305 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.813492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35625 (local address 127.0.97.1:54055)
0928 05:15:28.814574 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.814592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.814736 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.815168 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.815175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.815206 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:28.815774 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.815792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.816939 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.816946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.819028 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.819041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.820305 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.820312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.820507 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.820537 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.820652 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.820798 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":190,"thread_start_us":541,"threads_started":1}
I20260928 05:15:28.821405 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.813485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44657 (local address 127.0.97.1:33403)
0928 05:15:28.814493 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:28.814553 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:28.814788 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:28.815381 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:28.815386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:28.815419 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:28.815880 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.815892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.816994 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.817003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.819014 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:28.819028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.820367 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.820376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:28.820535 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.820560 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 05:15:28.820668 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:15:28.820804 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":36,"spinlock_wait_cycles":40704,"thread_start_us":502,"threads_started":1}
I20260928 05:15:28.822486 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a507cc7f1a44c0b8a789cffaaf7321" candidate_uuid: "8140d39711ab4f27b6eb326a3b69db5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29ba633ac0a485998336c98e62592f9" is_pre_election: true
I20260928 05:15:28.822718 17974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a507cc7f1a44c0b8a789cffaaf7321" candidate_uuid: "8140d39711ab4f27b6eb326a3b69db5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "760599a4c7da4cbdacf34ffbb854943d" is_pre_election: true
I20260928 05:15:28.823164 17841 raft_consensus.cc:2454] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8140d39711ab4f27b6eb326a3b69db5d in term 0.
I20260928 05:15:28.823244 17974 raft_consensus.cc:2454] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8140d39711ab4f27b6eb326a3b69db5d in term 0.
I20260928 05:15:28.823977 17641 leader_election.cc:302] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 760599a4c7da4cbdacf34ffbb854943d, 8140d39711ab4f27b6eb326a3b69db5d, c29ba633ac0a485998336c98e62592f9; no voters:
I20260928 05:15:28.824440 18045 raft_consensus.cc:2781] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:15:28.824606 18045 raft_consensus.cc:482] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:15:28.824713 18045 raft_consensus.cc:3037] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:28.826972 18045 raft_consensus.cc:504] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.827648 18045 leader_election.cc:288] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [CANDIDATE]: Term 1 election: Requested vote from peers c29ba633ac0a485998336c98e62592f9 (127.0.97.2:35625), 760599a4c7da4cbdacf34ffbb854943d (127.0.97.3:44657)
I20260928 05:15:28.828197 17974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a507cc7f1a44c0b8a789cffaaf7321" candidate_uuid: "8140d39711ab4f27b6eb326a3b69db5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "760599a4c7da4cbdacf34ffbb854943d"
I20260928 05:15:28.828239 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a507cc7f1a44c0b8a789cffaaf7321" candidate_uuid: "8140d39711ab4f27b6eb326a3b69db5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29ba633ac0a485998336c98e62592f9"
I20260928 05:15:28.828461 17974 raft_consensus.cc:3037] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:28.828478 17841 raft_consensus.cc:3037] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:28.830858 17841 raft_consensus.cc:2454] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8140d39711ab4f27b6eb326a3b69db5d in term 1.
I20260928 05:15:28.830938 17974 raft_consensus.cc:2454] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8140d39711ab4f27b6eb326a3b69db5d in term 1.
I20260928 05:15:28.831365 17641 leader_election.cc:302] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8140d39711ab4f27b6eb326a3b69db5d, c29ba633ac0a485998336c98e62592f9; no voters:
I20260928 05:15:28.831699 18045 raft_consensus.cc:2781] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:28.832237 18045 raft_consensus.cc:686] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 1 LEADER]: Becoming Leader. State: Replica: 8140d39711ab4f27b6eb326a3b69db5d, State: Running, Role: LEADER
I20260928 05:15:28.832818 18045 consensus_queue.cc:237] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } }
I20260928 05:15:28.837219 17563 catalog_manager.cc:5495] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8140d39711ab4f27b6eb326a3b69db5d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8140d39711ab4f27b6eb326a3b69db5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8140d39711ab4f27b6eb326a3b69db5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 } health_report { overall_health: UNKNOWN } } }
W20260928 05:15:28.850955 17753 tablet.cc:2369] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:28.854585 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.848533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.0.97.1:45667)
0928 05:15:28.848751 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.848758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.848832 (+ 74us) server_negotiation.cc:407] Connection header received
0928 05:15:28.849095 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.849099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.849199 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.849401 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:15:28.849814 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.850635 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.852230 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.852683 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.852789 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.853253 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:15:28.853793 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:15:28.854019 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.854080 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.854211 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 05:15:28.856487 17887 tablet.cc:2369] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:28.863795 17974 raft_consensus.cc:1261] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 1 FOLLOWER]: Refusing update from remote peer 8140d39711ab4f27b6eb326a3b69db5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:28.863814 17841 raft_consensus.cc:1261] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 1 FOLLOWER]: Refusing update from remote peer 8140d39711ab4f27b6eb326a3b69db5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:28.864830 18049 consensus_queue.cc:1035] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c29ba633ac0a485998336c98e62592f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:28.865242 18045 consensus_queue.cc:1035] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "760599a4c7da4cbdacf34ffbb854943d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:28.878162 18052 mvcc.cc:205] Tried to move back new op lower bound from 7334185078215262208 to 7334185078102929408. Current Snapshot: MvccSnapshot[applied={T|T < 7334185078215262208}]
I20260928 05:15:28.894557 17563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:28.894891 17563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:28.899009 17563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d93f2a735d740eb892e0387dfbf161f: Sending DeleteTablet for 3 replicas of tablet 07a507cc7f1a44c0b8a789cffaaf7321
I20260928 05:15:28.900460 17687 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a507cc7f1a44c0b8a789cffaaf7321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33576
I20260928 05:15:28.900461 17954 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a507cc7f1a44c0b8a789cffaaf7321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33058
I20260928 05:15:28.900784 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a507cc7f1a44c0b8a789cffaaf7321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33250
I20260928 05:15:28.901737 18063 tablet_replica.cc:330] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: stopping tablet replica
I20260928 05:15:28.902251 18063 raft_consensus.cc:2229] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:28.902776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17624
I20260928 05:15:28.902978 18063 raft_consensus.cc:2258] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:28.914762 18063 ts_tablet_manager.cc:1905] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:28.916939 18064 tablet_replica.cc:330] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: stopping tablet replica
I20260928 05:15:28.917687 18062 tablet_replica.cc:330] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: stopping tablet replica
I20260928 05:15:28.918243 18062 raft_consensus.cc:2229] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:15:28.918869 18064 raft_consensus.cc:2229] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:28.919636 18062 raft_consensus.cc:2258] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:28.920217 18064 raft_consensus.cc:2258] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:28.922767 18062 ts_tablet_manager.cc:1905] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:28.923849 18063 ts_tablet_manager.cc:1918] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:28.924036 18063 log.cc:1196] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/07a507cc7f1a44c0b8a789cffaaf7321
I20260928 05:15:28.924419 18064 ts_tablet_manager.cc:1905] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:28.924584 18063 ts_tablet_manager.cc:1939] T 07a507cc7f1a44c0b8a789cffaaf7321 P 760599a4c7da4cbdacf34ffbb854943d: Deleting consensus metadata
I20260928 05:15:28.926496 17550 catalog_manager.cc:4841] TS 760599a4c7da4cbdacf34ffbb854943d (127.0.97.3:44657): tablet 07a507cc7f1a44c0b8a789cffaaf7321 (table test-table [id=81d97ea83bb14ee2838319a0934b895c]) successfully deleted
Sep 28 05:15:28 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:15:28.935223 18064 ts_tablet_manager.cc:1918] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:28.935431 18064 log.cc:1196] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/07a507cc7f1a44c0b8a789cffaaf7321
I20260928 05:15:28.935477 18062 ts_tablet_manager.cc:1918] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:28.935770 18062 log.cc:1196] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/07a507cc7f1a44c0b8a789cffaaf7321
I20260928 05:15:28.935993 18064 ts_tablet_manager.cc:1939] T 07a507cc7f1a44c0b8a789cffaaf7321 P c29ba633ac0a485998336c98e62592f9: Deleting consensus metadata
I20260928 05:15:28.936496 18062 ts_tablet_manager.cc:1939] T 07a507cc7f1a44c0b8a789cffaaf7321 P 8140d39711ab4f27b6eb326a3b69db5d: Deleting consensus metadata
I20260928 05:15:28.938169 17548 catalog_manager.cc:4841] TS c29ba633ac0a485998336c98e62592f9 (127.0.97.2:35625): tablet 07a507cc7f1a44c0b8a789cffaaf7321 (table test-table [id=81d97ea83bb14ee2838319a0934b895c]) successfully deleted
I20260928 05:15:28.938488 17548 catalog_manager.cc:4841] TS 8140d39711ab4f27b6eb326a3b69db5d (127.0.97.1:45667): tablet 07a507cc7f1a44c0b8a789cffaaf7321 (table test-table [id=81d97ea83bb14ee2838319a0934b895c]) successfully deleted
I20260928 05:15:28.940752 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:28.904488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.0.97.1:45667)
0928 05:15:28.904732 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:28.904742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:28.919654 (+ 14912us) server_negotiation.cc:407] Connection header received
0928 05:15:28.919896 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:28.919902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:28.919994 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:28.920094 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:28.922677 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.923775 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.925823 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:28.926573 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:28.926692 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:28.932705 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:28.932726 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:28.932745 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:28.932807 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:28.937316 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.938453 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.938463 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.938468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.938694 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:28.939125 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:28.939129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:28.939133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:28.939516 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:28.939829 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:28.940154 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 05:15:28.940411 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:15:28.941382 17747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:29.041326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17624
I20260928 05:15:29.059979 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17758
Sep 28 05:15:29 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:15:29.075235 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:29.060739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.97.2:35625)
0928 05:15:29.060911 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:29.060915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:29.060933 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:15:29.060988 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:29.060992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:29.061066 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:29.061155 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:29.062433 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:29.063315 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:29.064557 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:29.064989 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:29.065108 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:29.069477 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:29.069493 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:29.069509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:29.069546 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:29.072553 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:29.073224 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:29.073232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:29.073234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:29.073397 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:29.073791 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:29.073794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:29.073796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:29.074225 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:29.074452 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:29.074819 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 05:15:29.074983 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:15:29.075901 17881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:29.164834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17758
I20260928 05:15:29.182267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17891
Sep 28 05:15:29 dist-test-slave-6718 krb5kdc[17518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:15:29.197577 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:29.183192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.0.97.3:44657)
0928 05:15:29.183363 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:29.183367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:29.183430 (+ 63us) server_negotiation.cc:407] Connection header received
0928 05:15:29.183606 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:29.183611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:29.183675 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:29.183760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:29.185030 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:29.185775 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:29.187168 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:29.187591 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:29.187699 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:29.192043 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:29.192062 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:29.192079 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:29.192122 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:29.195174 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:29.195831 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:29.195839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:29.195841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:29.195980 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:29.196275 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:29.196278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:29.196280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:29.196628 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:29.196854 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:29.197056 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 05:15:29.197219 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 05:15:29.198262 18014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:29.289565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17891
I20260928 05:15:29.306288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 17531
I20260928 05:15:29.307534 17593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:29.418668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 17531
2026-09-28T05:15:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4248 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T05:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:29Z Disabled control of system clock
I20260928 05:15:29.493527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46463
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:15:29.824864 18073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:29.825165 18073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:29.825309 18073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:29.832644 18073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:29.832763 18073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:29.832842 18073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:29.832949 18073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:29.833011 18073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:29.842367 18073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:29.843719 18073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:29.845584 18073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:29.857434 18081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:29.857532 18078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:29.857519 18079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:29.860021 18073 server_base.cc:1034] running on GCE node
I20260928 05:15:29.860859 18073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:29.862399 18073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:29.863651 18073 hybrid_clock.cc:648] HybridClock initialized: now 1790572529863612 us; error 58 us; skew 500 ppm
I20260928 05:15:29.864032 18073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:29.871290 18073 webserver.cc:460] Webserver started at http://127.0.97.62:37525/ using document root <none> and password file <none>
I20260928 05:15:29.872072 18073 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:29.872210 18073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:29.872538 18073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:29.875814 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "8b2273542d494a479688c8773936ed0e"
format_stamp: "Formatted at 2026-09-28 05:15:29 on dist-test-slave-6718"
server_key: "616f069b72c65bfe28304d617ab51ef4"
server_key_iv: "19f4c32605b475f284f700fd00308e5b"
server_key_version: "encryptionkey@0"
I20260928 05:15:29.876754 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b2273542d494a479688c8773936ed0e"
format_stamp: "Formatted at 2026-09-28 05:15:29 on dist-test-slave-6718"
server_key: "616f069b72c65bfe28304d617ab51ef4"
server_key_iv: "19f4c32605b475f284f700fd00308e5b"
server_key_version: "encryptionkey@0"
I20260928 05:15:29.882133 18073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:15:29.885823 18087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:29.887252 18073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:29.887442 18073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "8b2273542d494a479688c8773936ed0e"
format_stamp: "Formatted at 2026-09-28 05:15:29 on dist-test-slave-6718"
server_key: "616f069b72c65bfe28304d617ab51ef4"
server_key_iv: "19f4c32605b475f284f700fd00308e5b"
server_key_version: "encryptionkey@0"
I20260928 05:15:29.887660 18073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:29.914475 18073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:29.915439 18073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:29.915730 18073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:29.934806 18073 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46463
I20260928 05:15:29.934826 18138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46463 every 8 connection(s)
I20260928 05:15:29.936218 18073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:29.936663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18073
I20260928 05:15:29.936852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:29.937477 388 external_mini_cluster.cc:1428] Setting key 4b452cb158ec71d4021a674b509f34de
I20260928 05:15:29.941246 18139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:29.951670 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:29.939580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.0.97.62:46463)
0928 05:15:29.940712 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:29.940723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:29.940768 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:15:29.942689 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:29.942714 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:29.943206 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:29.943641 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:29.944268 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:29.945615 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:29.947162 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:29.947865 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:29.948071 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:29.949235 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:29.949245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:29.949260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:29.949849 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:29.949920 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:29.949941 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:29.950134 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:29.950354 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 05:15:29.950718 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":400,"threads_started":1}
I20260928 05:15:29.952677 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e: Bootstrap starting.
I20260928 05:15:29.956028 18139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:29.956966 18139 log.cc:824] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:29.959992 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e: No bootstrap required, opened a new log
I20260928 05:15:29.967545 18139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2273542d494a479688c8773936ed0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46463 } }
I20260928 05:15:29.967886 18139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:29.967967 18139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2273542d494a479688c8773936ed0e, State: Initialized, Role: FOLLOWER
I20260928 05:15:29.968529 18139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2273542d494a479688c8773936ed0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46463 } }
I20260928 05:15:29.968726 18139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:29.968813 18139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:29.968942 18139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:29.971050 18139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2273542d494a479688c8773936ed0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46463 } }
I20260928 05:15:29.971547 18139 leader_election.cc:302] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b2273542d494a479688c8773936ed0e; no voters:
I20260928 05:15:29.972132 18139 leader_election.cc:288] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:29.972302 18144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:29.973030 18144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [term 1 LEADER]: Becoming Leader. State: Replica: 8b2273542d494a479688c8773936ed0e, State: Running, Role: LEADER
I20260928 05:15:29.973632 18144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2273542d494a479688c8773936ed0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46463 } }
I20260928 05:15:29.974413 18139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:29.976576 18145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b2273542d494a479688c8773936ed0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2273542d494a479688c8773936ed0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46463 } } }
I20260928 05:15:29.977068 18145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:29.976600 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b2273542d494a479688c8773936ed0e. Latest consensus state: current_term: 1 leader_uuid: "8b2273542d494a479688c8773936ed0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2273542d494a479688c8773936ed0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46463 } } }
I20260928 05:15:29.977954 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:29.983515 18151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:29.987859 18151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:29.996307 18151 catalog_manager.cc:1347] Generated new cluster ID: 835dc23e63a8442eb0bf15d0438940a4
I20260928 05:15:29.996428 18151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:30.009369 18151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:30.010977 18151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:30.020066 18151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e: Generated new TSK 0
I20260928 05:15:30.021106 18151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:30.038928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46463
--builtin_ntp_servers=127.0.97.20:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:15:30.373871 18163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:30.374217 18163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:30.374379 18163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:30.381981 18163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:30.382125 18163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:30.382305 18163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:30.391637 18163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:30.392973 18163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:30.394858 18163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:30.407701 18168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:30.407724 18169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:30.407796 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:30.408439 18163 server_base.cc:1034] running on GCE node
I20260928 05:15:30.409097 18163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:30.410044 18163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:30.411242 18163 hybrid_clock.cc:648] HybridClock initialized: now 1790572530411192 us; error 44 us; skew 500 ppm
I20260928 05:15:30.411568 18163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:30.413956 18163 webserver.cc:460] Webserver started at http://127.0.97.1:39427/ using document root <none> and password file <none>
I20260928 05:15:30.414718 18163 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:30.414830 18163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:30.415169 18163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:30.418135 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "97961220d4294d3f8b717a946ff24b5d"
format_stamp: "Formatted at 2026-09-28 05:15:30 on dist-test-slave-6718"
server_key: "c41f6358b38287fcba3eb8de76c4e526"
server_key_iv: "13510839474f3e114fb26ef61c50f252"
server_key_version: "encryptionkey@0"
I20260928 05:15:30.419071 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "97961220d4294d3f8b717a946ff24b5d"
format_stamp: "Formatted at 2026-09-28 05:15:30 on dist-test-slave-6718"
server_key: "c41f6358b38287fcba3eb8de76c4e526"
server_key_iv: "13510839474f3e114fb26ef61c50f252"
server_key_version: "encryptionkey@0"
I20260928 05:15:30.424516 18163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:15:30.428092 18177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:30.429499 18163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:15:30.429636 18163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "97961220d4294d3f8b717a946ff24b5d"
format_stamp: "Formatted at 2026-09-28 05:15:30 on dist-test-slave-6718"
server_key: "c41f6358b38287fcba3eb8de76c4e526"
server_key_iv: "13510839474f3e114fb26ef61c50f252"
server_key_version: "encryptionkey@0"
I20260928 05:15:30.429818 18163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:30.448154 18163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:30.448912 18163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:30.449141 18163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:30.450032 18163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:30.451833 18163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:30.451956 18163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:30.452117 18163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:30.452196 18163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:30.486043 18163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41921
I20260928 05:15:30.486100 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41921 every 8 connection(s)
I20260928 05:15:30.487437 18163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:30.493896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18163
I20260928 05:15:30.494093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:15:30.494570 388 external_mini_cluster.cc:1428] Setting key ee35497299a8add6901492f45ceecf0c
I20260928 05:15:30.497219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46463
--builtin_ntp_servers=127.0.97.20:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:15:30.500857 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:30.489976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54687 (local address 127.0.97.62:46463)
0928 05:15:30.490474 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:30.490480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:30.491744 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 05:15:30.493591 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:30.493599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:30.493696 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:30.493829 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:30.495003 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.495898 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.497672 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.498449 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.498614 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:30.499923 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:30.499931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:30.499939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:30.499963 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:30.500011 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:30.500029 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:30.500120 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:30.500340 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 05:15:30.500446 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260928 05:15:30.501644 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:30.490647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46463 (local address 127.0.97.1:54687)
0928 05:15:30.491598 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:30.491639 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:30.493324 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:30.494062 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:30.494070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:30.494183 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:30.494833 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:30.494852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.496051 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.496056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:30.497541 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:30.497553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.498687 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.498692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:30.498879 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:30.499605 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:30.499607 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:30.499648 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:30.499727 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:30.500124 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:30.500132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:30.500137 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 05:15:30.500279 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 05:15:30.500567 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":273,"threads_started":1}
I20260928 05:15:30.503259 18290 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46463
I20260928 05:15:30.503590 18290 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:30.504500 18290 heartbeater.cc:511] Master 127.0.97.62:46463 requested a full tablet report, sending...
I20260928 05:15:30.506912 18104 ts_manager.cc:194] Registered new tserver with Master: 97961220d4294d3f8b717a946ff24b5d (127.0.97.1:41921)
I20260928 05:15:30.508383 18104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54687
W20260928 05:15:30.831718 18294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:30.832074 18294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:30.832242 18294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:30.839740 18294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:30.839908 18294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:30.840096 18294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:30.849375 18294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:30.850754 18294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:30.852504 18294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:30.865739 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:30.865677 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:30.865751 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:30.866521 18294 server_base.cc:1034] running on GCE node
I20260928 05:15:30.867213 18294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:30.868283 18294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:30.869515 18294 hybrid_clock.cc:648] HybridClock initialized: now 1790572530869447 us; error 65 us; skew 500 ppm
I20260928 05:15:30.869840 18294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:30.872254 18294 webserver.cc:460] Webserver started at http://127.0.97.2:38391/ using document root <none> and password file <none>
I20260928 05:15:30.873040 18294 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:30.873193 18294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:30.873523 18294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:30.876708 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "53de0defcd2646438648299e7c507793"
format_stamp: "Formatted at 2026-09-28 05:15:30 on dist-test-slave-6718"
server_key: "3046d3e514b8a478505c039bf5f6e02e"
server_key_iv: "bc6899aef9687e47a021f04e24f2f2bf"
server_key_version: "encryptionkey@0"
I20260928 05:15:30.877532 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "53de0defcd2646438648299e7c507793"
format_stamp: "Formatted at 2026-09-28 05:15:30 on dist-test-slave-6718"
server_key: "3046d3e514b8a478505c039bf5f6e02e"
server_key_iv: "bc6899aef9687e47a021f04e24f2f2bf"
server_key_version: "encryptionkey@0"
I20260928 05:15:30.883093 18294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:15:30.886613 18308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:30.887995 18294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:30.888185 18294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "53de0defcd2646438648299e7c507793"
format_stamp: "Formatted at 2026-09-28 05:15:30 on dist-test-slave-6718"
server_key: "3046d3e514b8a478505c039bf5f6e02e"
server_key_iv: "bc6899aef9687e47a021f04e24f2f2bf"
server_key_version: "encryptionkey@0"
I20260928 05:15:30.888427 18294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:30.902679 18294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:30.903611 18294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:30.903918 18294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:30.904820 18294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:30.906216 18294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:30.906318 18294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:30.906409 18294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:30.906464 18294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:30.939988 18294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45601
I20260928 05:15:30.940028 18420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45601 every 8 connection(s)
I20260928 05:15:30.941713 18294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:30.944939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18294
I20260928 05:15:30.945119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:15:30.945737 388 external_mini_cluster.cc:1428] Setting key 1a6cf9cf3e928e527a7629b1dfdcca04
I20260928 05:15:30.948474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46463
--builtin_ntp_servers=127.0.97.20:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:15:30.957664 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:30.944555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38611 (local address 127.0.97.62:46463)
0928 05:15:30.944731 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:30.944736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:30.946558 (+ 1822us) server_negotiation.cc:407] Connection header received
0928 05:15:30.949100 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:30.949109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:30.949211 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:30.949322 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:30.950697 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.951900 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.954230 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.954995 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.955139 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:30.956823 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:30.956829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:30.956832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:30.956852 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:30.956884 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:30.956897 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:30.957002 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:30.957260 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 05:15:30.957363 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:15:30.958905 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:30.945198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46463 (local address 127.0.97.2:38611)
0928 05:15:30.946366 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:30.946411 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:30.948784 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:30.949574 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:30.949585 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:30.949711 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:30.950481 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:30.950507 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.952069 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.952075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:30.954068 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:30.954082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:30.955227 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:30.955233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:30.955533 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:30.956446 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:30.956448 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:30.956491 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:30.956575 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:30.957002 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:30.957014 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:30.957025 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:15:30.957238 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 05:15:30.957671 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":329,"threads_started":1}
I20260928 05:15:30.960448 18421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46463
I20260928 05:15:30.960764 18421 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:30.961535 18421 heartbeater.cc:511] Master 127.0.97.62:46463 requested a full tablet report, sending...
I20260928 05:15:30.963106 18104 ts_manager.cc:194] Registered new tserver with Master: 53de0defcd2646438648299e7c507793 (127.0.97.2:45601)
I20260928 05:15:30.964341 18104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38611
W20260928 05:15:31.280464 18424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:31.280843 18424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:31.280942 18424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:31.289275 18424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:31.289400 18424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:31.289525 18424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:31.300112 18424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:31.301541 18424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:31.303377 18424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:31.316963 18432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:31.316982 18430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:31.317163 18429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:31.317862 18424 server_base.cc:1034] running on GCE node
I20260928 05:15:31.318463 18424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:31.319520 18424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:31.320776 18424 hybrid_clock.cc:648] HybridClock initialized: now 1790572531320714 us; error 56 us; skew 500 ppm
I20260928 05:15:31.321154 18424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:31.324012 18424 webserver.cc:460] Webserver started at http://127.0.97.3:46723/ using document root <none> and password file <none>
I20260928 05:15:31.324752 18424 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:31.324898 18424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:31.325227 18424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:31.328240 18424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "b63f203606824c23ac68b333460524b4"
format_stamp: "Formatted at 2026-09-28 05:15:31 on dist-test-slave-6718"
server_key: "61728db240e569db312347be5b77fc65"
server_key_iv: "fdd480d3f60731785f06837b1534ec4c"
server_key_version: "encryptionkey@0"
I20260928 05:15:31.329068 18424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b63f203606824c23ac68b333460524b4"
format_stamp: "Formatted at 2026-09-28 05:15:31 on dist-test-slave-6718"
server_key: "61728db240e569db312347be5b77fc65"
server_key_iv: "fdd480d3f60731785f06837b1534ec4c"
server_key_version: "encryptionkey@0"
I20260928 05:15:31.334638 18424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:15:31.338179 18438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:31.339614 18424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:31.339805 18424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "b63f203606824c23ac68b333460524b4"
format_stamp: "Formatted at 2026-09-28 05:15:31 on dist-test-slave-6718"
server_key: "61728db240e569db312347be5b77fc65"
server_key_iv: "fdd480d3f60731785f06837b1534ec4c"
server_key_version: "encryptionkey@0"
I20260928 05:15:31.340044 18424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:31.375097 18424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:31.376085 18424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:31.376394 18424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:31.377470 18424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:31.379096 18424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:31.379223 18424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:31.379359 18424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:31.379470 18424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:31.413615 18424 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44687
I20260928 05:15:31.413741 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44687 every 8 connection(s)
I20260928 05:15:31.415069 18424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:31.415704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18424
I20260928 05:15:31.415891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:15:31.416559 388 external_mini_cluster.cc:1428] Setting key 4b58a7986acf43f11b096d94715dd64f
I20260928 05:15:31.428925 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.418582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50911 (local address 127.0.97.62:46463)
0928 05:15:31.418763 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.418769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.420678 (+ 1909us) server_negotiation.cc:407] Connection header received
0928 05:15:31.422642 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.422646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.422748 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.422850 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:31.423982 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.424730 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.426351 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.426831 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.426928 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.428211 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:31.428216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:31.428219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:31.428236 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:31.428265 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:31.428277 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:31.428381 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.428603 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.428689 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:15:31.430019 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.419294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46463 (local address 127.0.97.3:50911)
0928 05:15:31.420504 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:31.420552 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:31.422300 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:31.423093 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:31.423102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:31.423196 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:31.423832 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.423849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.424866 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.424870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.426206 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.426218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.427028 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.427033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.427241 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.427913 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:31.427916 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:31.427955 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:31.428037 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:31.428390 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:31.428398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:31.428404 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:15:31.428579 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 05:15:31.428880 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":372,"threads_started":1}
I20260928 05:15:31.431666 18551 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46463
I20260928 05:15:31.432034 18551 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:31.432878 18551 heartbeater.cc:511] Master 127.0.97.62:46463 requested a full tablet report, sending...
I20260928 05:15:31.434439 18103 ts_manager.cc:194] Registered new tserver with Master: b63f203606824c23ac68b333460524b4 (127.0.97.3:44687)
I20260928 05:15:31.435385 18103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:50911
I20260928 05:15:31.445595 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:15:31.455668 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.449769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.0.97.62:46463)
0928 05:15:31.450010 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.450018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.450308 (+ 290us) server_negotiation.cc:407] Connection header received
0928 05:15:31.450509 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.450515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.450648 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.450741 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:31.451206 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.451910 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.453278 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.453706 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.453833 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.454964 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:31.454970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:31.454975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:31.455000 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:31.455032 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:31.455044 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:31.455139 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.455276 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.455377 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 05:15:31.459769 18103 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:15:31.466656 18103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:15:31.500013 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.478934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.0.97.2:45601)
0928 05:15:31.482918 (+ 3984us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.482930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.482967 (+ 37us) server_negotiation.cc:407] Connection header received
0928 05:15:31.483085 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.483096 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.483403 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.483625 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:31.484758 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.493986 (+ 9228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.497240 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.498780 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.498930 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.499191 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.499493 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.499675 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2563,"thread_start_us":2456,"threads_started":1}
I20260928 05:15:31.500690 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.478568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45601 (local address 127.0.0.1:49004)
0928 05:15:31.480870 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:31.480907 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:31.481125 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:31.483956 (+ 2831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:31.483967 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:31.484008 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:31.484516 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.484532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.494203 (+ 9671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.494212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.496999 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.497013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.498991 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.498997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.499212 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.499292 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 05:15:31.499421 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 05:15:31.499620 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1987,"mutex_wait_us":37,"thread_start_us":15593,"threads_started":1}
I20260928 05:15:31.501157 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.478672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50902)
0928 05:15:31.491758 (+ 13086us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:31.491783 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:31.491962 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:31.493040 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:31.493045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:31.493073 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:31.493525 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.493538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.495985 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.495990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.497935 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.497947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.500319 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.500329 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.500495 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.500519 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:15:31.500630 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 05:15:31.500751 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12847,"mutex_wait_us":1,"thread_start_us":1748,"threads_started":1}
I20260928 05:15:31.501736 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.479358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.0.97.1:41921)
0928 05:15:31.490693 (+ 11335us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.490705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.491890 (+ 1185us) server_negotiation.cc:407] Connection header received
0928 05:15:31.492136 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.492144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.492474 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.492779 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:31.493715 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.495821 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.498129 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.499506 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.499633 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.499882 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.501368 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.501493 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11126,"thread_start_us":10986,"threads_started":1}
I20260928 05:15:31.502758 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.481263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44687 (local address 127.0.0.1:51512)
0928 05:15:31.494563 (+ 13300us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:31.494582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:31.494713 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:31.495657 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:31.495661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:31.495687 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:31.496128 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.496143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.498058 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.498066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.500207 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.500225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.502051 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.502056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.502201 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.502230 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 05:15:31.502320 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 05:15:31.502459 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13143,"thread_start_us":10058,"threads_started":1}
I20260928 05:15:31.502923 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.481788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.0.97.3:44687)
0928 05:15:31.491113 (+ 9325us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.491124 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.494703 (+ 3579us) server_negotiation.cc:407] Connection header received
0928 05:15:31.494866 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.494875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.495131 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.495324 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:31.496359 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.497902 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.500468 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.501885 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.502022 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.502292 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.502354 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.502516 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9113,"thread_start_us":8953,"threads_started":1}
I20260928 05:15:31.505268 18356 tablet_service.cc:1467] Processing CreateTablet for tablet a43bd70887ca44b2a4aef2131fd6ec53 (DEFAULT_TABLE table=test-table [id=84d09f3fc7f0421c85da76c6b8f6ff05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:31.506013 18225 tablet_service.cc:1467] Processing CreateTablet for tablet a43bd70887ca44b2a4aef2131fd6ec53 (DEFAULT_TABLE table=test-table [id=84d09f3fc7f0421c85da76c6b8f6ff05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:31.506250 18486 tablet_service.cc:1467] Processing CreateTablet for tablet a43bd70887ca44b2a4aef2131fd6ec53 (DEFAULT_TABLE table=test-table [id=84d09f3fc7f0421c85da76c6b8f6ff05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:31.506798 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43bd70887ca44b2a4aef2131fd6ec53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:31.507524 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43bd70887ca44b2a4aef2131fd6ec53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:31.507643 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43bd70887ca44b2a4aef2131fd6ec53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:31.512168 18290 heartbeater.cc:503] Master 127.0.97.62:46463 was elected leader, sending a full tablet report...
I20260928 05:15:31.519618 18569 tablet_bootstrap.cc:492] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Bootstrap starting.
I20260928 05:15:31.523183 18569 tablet_bootstrap.cc:654] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:31.524423 18569 log.cc:824] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:31.527387 18569 tablet_bootstrap.cc:492] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: No bootstrap required, opened a new log
I20260928 05:15:31.527654 18569 ts_tablet_manager.cc:1397] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:15:31.533710 18571 tablet_bootstrap.cc:492] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: Bootstrap starting.
I20260928 05:15:31.534392 18570 tablet_bootstrap.cc:492] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Bootstrap starting.
I20260928 05:15:31.535804 18569 raft_consensus.cc:348] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.536276 18569 raft_consensus.cc:374] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:31.536398 18569 raft_consensus.cc:729] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63f203606824c23ac68b333460524b4, State: Initialized, Role: FOLLOWER
I20260928 05:15:31.537022 18569 consensus_queue.cc:260] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.537261 18571 tablet_bootstrap.cc:654] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:31.537992 18570 tablet_bootstrap.cc:654] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:31.538159 18551 heartbeater.cc:503] Master 127.0.97.62:46463 was elected leader, sending a full tablet report...
I20260928 05:15:31.538589 18571 log.cc:824] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:31.539201 18570 log.cc:824] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:31.539744 18569 ts_tablet_manager.cc:1428] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 05:15:31.545293 18571 tablet_bootstrap.cc:492] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: No bootstrap required, opened a new log
I20260928 05:15:31.545310 18570 tablet_bootstrap.cc:492] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: No bootstrap required, opened a new log
I20260928 05:15:31.545573 18571 ts_tablet_manager.cc:1397] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260928 05:15:31.545578 18570 ts_tablet_manager.cc:1397] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20260928 05:15:31.552940 18570 raft_consensus.cc:348] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.552942 18571 raft_consensus.cc:348] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.553316 18571 raft_consensus.cc:374] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:31.553324 18570 raft_consensus.cc:374] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:31.553413 18571 raft_consensus.cc:729] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97961220d4294d3f8b717a946ff24b5d, State: Initialized, Role: FOLLOWER
I20260928 05:15:31.553434 18570 raft_consensus.cc:729] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53de0defcd2646438648299e7c507793, State: Initialized, Role: FOLLOWER
I20260928 05:15:31.554026 18571 consensus_queue.cc:260] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.554028 18570 consensus_queue.cc:260] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.555609 18570 ts_tablet_manager.cc:1428] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260928 05:15:31.555771 18571 ts_tablet_manager.cc:1428] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260928 05:15:31.555989 18421 heartbeater.cc:503] Master 127.0.97.62:46463 was elected leader, sending a full tablet report...
W20260928 05:15:31.667478 18552 tablet.cc:2369] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:15:31.693871 18422 tablet.cc:2369] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:15:31.739876 18291 tablet.cc:2369] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:31.756035 18573 raft_consensus.cc:482] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:31.756382 18573 raft_consensus.cc:504] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.758100 18573 leader_election.cc:288] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53de0defcd2646438648299e7c507793 (127.0.97.2:45601), 97961220d4294d3f8b717a946ff24b5d (127.0.97.1:41921)
I20260928 05:15:31.765005 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.758671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45601 (local address 127.0.97.3:46021)
0928 05:15:31.758897 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:31.758918 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:31.759078 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:31.759564 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:31.759570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:31.759602 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:31.759829 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.759853 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.761210 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.761214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.763190 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.763206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.764453 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.764457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.764575 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.764601 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:15:31.764676 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:15:31.764776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":1}
I20260928 05:15:31.765167 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.758666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46021 (local address 127.0.97.2:45601)
0928 05:15:31.758889 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.758895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.759017 (+ 122us) server_negotiation.cc:407] Connection header received
0928 05:15:31.759227 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.759236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.759353 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.759488 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:31.760151 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.761068 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.763357 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.764269 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.764409 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.764502 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.764725 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.764879 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:15:31.766079 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.758751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41069 (local address 127.0.97.1:41921)
0928 05:15:31.758979 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.758986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.759894 (+ 908us) server_negotiation.cc:407] Connection header received
0928 05:15:31.760165 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.760171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.760282 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.760429 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:31.760993 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.761733 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.763947 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.765101 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.765234 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.765344 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.765730 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.765855 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:15:31.766162 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.758677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.97.3:41069)
0928 05:15:31.759781 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:31.759799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:31.759921 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:31.760520 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:31.760524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:31.760546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:31.760813 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.760829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.761920 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.761925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.763757 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:31.763767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.765434 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.765438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:31.765558 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.765582 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 05:15:31.765690 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:15:31.765823 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":238,"thread_start_us":482,"threads_started":1}
I20260928 05:15:31.766131 18376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43bd70887ca44b2a4aef2131fd6ec53" candidate_uuid: "b63f203606824c23ac68b333460524b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53de0defcd2646438648299e7c507793" is_pre_election: true
I20260928 05:15:31.766855 18376 raft_consensus.cc:2454] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63f203606824c23ac68b333460524b4 in term 0.
I20260928 05:15:31.767225 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43bd70887ca44b2a4aef2131fd6ec53" candidate_uuid: "b63f203606824c23ac68b333460524b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97961220d4294d3f8b717a946ff24b5d" is_pre_election: true
I20260928 05:15:31.767696 18440 leader_election.cc:302] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53de0defcd2646438648299e7c507793, b63f203606824c23ac68b333460524b4; no voters:
I20260928 05:15:31.767834 18245 raft_consensus.cc:2454] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b63f203606824c23ac68b333460524b4 in term 0.
I20260928 05:15:31.768114 18573 raft_consensus.cc:2781] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:15:31.768267 18573 raft_consensus.cc:482] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:15:31.768430 18573 raft_consensus.cc:3037] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:31.770694 18573 raft_consensus.cc:504] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.771322 18573 leader_election.cc:288] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [CANDIDATE]: Term 1 election: Requested vote from peers 53de0defcd2646438648299e7c507793 (127.0.97.2:45601), 97961220d4294d3f8b717a946ff24b5d (127.0.97.1:41921)
I20260928 05:15:31.771852 18376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43bd70887ca44b2a4aef2131fd6ec53" candidate_uuid: "b63f203606824c23ac68b333460524b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53de0defcd2646438648299e7c507793"
I20260928 05:15:31.771852 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43bd70887ca44b2a4aef2131fd6ec53" candidate_uuid: "b63f203606824c23ac68b333460524b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97961220d4294d3f8b717a946ff24b5d"
I20260928 05:15:31.772063 18245 raft_consensus.cc:3037] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:31.772063 18376 raft_consensus.cc:3037] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:31.775336 18245 raft_consensus.cc:2454] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63f203606824c23ac68b333460524b4 in term 1.
I20260928 05:15:31.775329 18376 raft_consensus.cc:2454] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63f203606824c23ac68b333460524b4 in term 1.
I20260928 05:15:31.775728 18440 leader_election.cc:302] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53de0defcd2646438648299e7c507793, b63f203606824c23ac68b333460524b4; no voters:
I20260928 05:15:31.776037 18573 raft_consensus.cc:2781] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:31.776497 18573 raft_consensus.cc:686] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 1 LEADER]: Becoming Leader. State: Replica: b63f203606824c23ac68b333460524b4, State: Running, Role: LEADER
I20260928 05:15:31.777040 18573 consensus_queue.cc:237] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } }
I20260928 05:15:31.781683 18104 catalog_manager.cc:5495] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b63f203606824c23ac68b333460524b4 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b63f203606824c23ac68b333460524b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b63f203606824c23ac68b333460524b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44687 } health_report { overall_health: HEALTHY } } }
I20260928 05:15:31.861193 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.855541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.0.97.3:44687)
0928 05:15:31.855746 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.855750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.855768 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:15:31.855937 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.855941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.856086 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.856276 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:15:31.856772 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.857626 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.859097 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.859516 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.859667 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.860093 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:15:31.860648 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:15:31.860827 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.860877 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.860942 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 05:15:31.869783 18376 raft_consensus.cc:1261] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 1 FOLLOWER]: Refusing update from remote peer b63f203606824c23ac68b333460524b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:31.869783 18245 raft_consensus.cc:1261] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 1 FOLLOWER]: Refusing update from remote peer b63f203606824c23ac68b333460524b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:31.870970 18579 consensus_queue.cc:1035] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53de0defcd2646438648299e7c507793" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:31.871387 18573 consensus_queue.cc:1035] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97961220d4294d3f8b717a946ff24b5d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:31.887218 18583 mvcc.cc:205] Tried to move back new op lower bound from 7334185090527125504 to 7334185090163101696. Current Snapshot: MvccSnapshot[applied={T|T < 7334185090527125504}]
I20260928 05:15:31.892364 18584 mvcc.cc:205] Tried to move back new op lower bound from 7334185090527125504 to 7334185090163101696. Current Snapshot: MvccSnapshot[applied={T|T < 7334185090527125504}]
I20260928 05:15:31.900454 18104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:31.900730 18104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:31.904384 18104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b2273542d494a479688c8773936ed0e: Sending DeleteTablet for 3 replicas of tablet a43bd70887ca44b2a4aef2131fd6ec53
I20260928 05:15:31.905612 18356 tablet_service.cc:1514] Processing DeleteTablet for tablet a43bd70887ca44b2a4aef2131fd6ec53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:31 UTC) from {username='slave'} at 127.0.0.1:49004
I20260928 05:15:31.905897 18225 tablet_service.cc:1514] Processing DeleteTablet for tablet a43bd70887ca44b2a4aef2131fd6ec53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:31 UTC) from {username='slave'} at 127.0.0.1:50902
I20260928 05:15:31.907222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18163
I20260928 05:15:31.907205 18486 tablet_service.cc:1514] Processing DeleteTablet for tablet a43bd70887ca44b2a4aef2131fd6ec53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:31 UTC) from {username='slave'} at 127.0.0.1:51512
I20260928 05:15:31.915427 18592 tablet_replica.cc:330] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: stopping tablet replica
I20260928 05:15:31.916018 18592 raft_consensus.cc:2229] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:31.916628 18592 raft_consensus.cc:2258] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:31.919075 18592 ts_tablet_manager.cc:1905] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:31.920964 18594 tablet_replica.cc:330] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: stopping tablet replica
I20260928 05:15:31.921584 18594 raft_consensus.cc:2229] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:15:31.922545 18594 raft_consensus.cc:2258] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:31.926685 18592 ts_tablet_manager.cc:1918] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:31.926914 18592 log.cc:1196] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/a43bd70887ca44b2a4aef2131fd6ec53
I20260928 05:15:31.927592 18592 ts_tablet_manager.cc:1939] T a43bd70887ca44b2a4aef2131fd6ec53 P 53de0defcd2646438648299e7c507793: Deleting consensus metadata
I20260928 05:15:31.927874 18594 ts_tablet_manager.cc:1905] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:31.929283 18593 tablet_replica.cc:330] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d: stopping tablet replica
I20260928 05:15:31.929605 18089 catalog_manager.cc:4841] TS 53de0defcd2646438648299e7c507793 (127.0.97.2:45601): tablet a43bd70887ca44b2a4aef2131fd6ec53 (table test-table [id=84d09f3fc7f0421c85da76c6b8f6ff05]) successfully deleted
I20260928 05:15:31.929857 18593 raft_consensus.cc:2229] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:31.930292 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:31.908065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.0.97.1:41921)
0928 05:15:31.908244 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:31.908250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:31.915180 (+ 6930us) server_negotiation.cc:407] Connection header received
0928 05:15:31.915627 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:31.915638 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:31.915771 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:31.915871 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:31.916670 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.918070 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.922818 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:31.923839 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:31.923978 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:31.928403 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:31.928420 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:31.928445 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:31.929365 (+ 920us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:31.929430 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:31.929454 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:31.929552 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:31.929795 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 05:15:31.929941 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:15:31.930503 18593 raft_consensus.cc:2258] T a43bd70887ca44b2a4aef2131fd6ec53 P 97961220d4294d3f8b717a946ff24b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:31.931550 18285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:31.942126 18594 ts_tablet_manager.cc:1918] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:31.942413 18594 log.cc:1196] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/a43bd70887ca44b2a4aef2131fd6ec53
I20260928 05:15:31.943040 18594 ts_tablet_manager.cc:1939] T a43bd70887ca44b2a4aef2131fd6ec53 P b63f203606824c23ac68b333460524b4: Deleting consensus metadata
I20260928 05:15:31.944926 18090 catalog_manager.cc:4841] TS b63f203606824c23ac68b333460524b4 (127.0.97.3:44687): tablet a43bd70887ca44b2a4aef2131fd6ec53 (table test-table [id=84d09f3fc7f0421c85da76c6b8f6ff05]) successfully deleted
I20260928 05:15:32.031924 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18163
W20260928 05:15:32.053862 18090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41921 (error 108)
I20260928 05:15:32.055045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18294
I20260928 05:15:32.056447 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.055514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50920)
0928 05:15:32.056083 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.056235 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":335,"threads_started":1}
W20260928 05:15:32.056752 18090 catalog_manager.cc:4584] TS 97961220d4294d3f8b717a946ff24b5d (127.0.97.1:41921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a43bd70887ca44b2a4aef2131fd6ec53: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
I20260928 05:15:32.063730 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.056104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.0.97.2:45601)
0928 05:15:32.056301 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:32.056307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:32.056590 (+ 283us) server_negotiation.cc:407] Connection header received
0928 05:15:32.056664 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:32.056669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:32.056772 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:32.056886 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:32.057334 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:32.058442 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:32.059753 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:32.060448 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:32.060577 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:32.061854 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:32.061866 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:32.061885 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:32.062876 (+ 991us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:32.062940 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:32.062964 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:32.063079 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:32.063246 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 05:15:32.063370 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:15:32.064534 18416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:32.110853 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.110102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50924)
0928 05:15:32.110447 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.110650 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260928 05:15:32.114270 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.113239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50940)
0928 05:15:32.113859 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.113987 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374}
I20260928 05:15:32.175199 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.174761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50946)
0928 05:15:32.174941 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.175062 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 05:15:32.176393 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.176049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50950)
0928 05:15:32.176183 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.176262 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 05:15:32.206991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18294
I20260928 05:15:32.230149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18424
I20260928 05:15:32.238049 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.231017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.0.97.3:44687)
0928 05:15:32.231199 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:32.231207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:32.231292 (+ 85us) server_negotiation.cc:407] Connection header received
0928 05:15:32.231490 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:32.231495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:32.231579 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:32.231665 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:32.232125 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:32.232913 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:32.234903 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:32.235352 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:32.235456 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:32.236647 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:32.236655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:32.236671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:32.237236 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:32.237287 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:32.237306 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:32.237425 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:32.237597 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 05:15:32.237681 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:15:32.238824 18546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:32.257174 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.256738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50962)
0928 05:15:32.256950 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.257036 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:15:32.258459 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:32.258040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41921 (local address 127.0.0.1:50974)
0928 05:15:32.258233 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:32.258301 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 05:15:32.362179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18424
I20260928 05:15:32.381945 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18073
I20260928 05:15:32.383330 18134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:32.494035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18073
2026-09-28T05:15:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3076 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:32 dist-test-slave-6718 krb5kdc[18604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:32 dist-test-slave-6718 krb5kdc[18604](info): set up 2 sockets
Sep 28 05:15:32 dist-test-slave-6718 krb5kdc[18604](info): commencing operation
krb5kdc: starting...
W20260928 05:15:33.626699 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:33 dist-test-slave-6718 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:15:33.875470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40501
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:34.207710 18617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:34.208006 18617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:34.208161 18617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:34.215581 18617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:34.215694 18617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:34.215778 18617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:34.215884 18617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:34.215948 18617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:34.225744 18617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:34.227069 18617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:34.228334 18617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:34.239579 18622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:34.239647 18623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:34.239935 18625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:34.240963 18617 server_base.cc:1034] running on GCE node
I20260928 05:15:34.241753 18617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:34.243178 18617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:34.244375 18617 hybrid_clock.cc:648] HybridClock initialized: now 1790572534244298 us; error 63 us; skew 500 ppm
I20260928 05:15:34.244652 18617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:34 dist-test-slave-6718 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:34.247758 18617 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:15:34.249292 18617 webserver.cc:460] Webserver started at http://127.0.97.62:39445/ using document root <none> and password file <none>
I20260928 05:15:34.249979 18617 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:34.250110 18617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:34.250450 18617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:34.253427 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "e7834258e4bf4e55b2024ce5a00f3113"
format_stamp: "Formatted at 2026-09-28 05:15:34 on dist-test-slave-6718"
server_key: "126d6c06518fdd8409bf1b0ffd1d205a"
server_key_iv: "ca5b193fbb7b1653ad4f9236a0f41d1d"
server_key_version: "encryptionkey@0"
I20260928 05:15:34.254199 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7834258e4bf4e55b2024ce5a00f3113"
format_stamp: "Formatted at 2026-09-28 05:15:34 on dist-test-slave-6718"
server_key: "126d6c06518fdd8409bf1b0ffd1d205a"
server_key_iv: "ca5b193fbb7b1653ad4f9236a0f41d1d"
server_key_version: "encryptionkey@0"
I20260928 05:15:34.259627 18617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:15:34.263204 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:34.264604 18617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:15:34.264806 18617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "e7834258e4bf4e55b2024ce5a00f3113"
format_stamp: "Formatted at 2026-09-28 05:15:34 on dist-test-slave-6718"
server_key: "126d6c06518fdd8409bf1b0ffd1d205a"
server_key_iv: "ca5b193fbb7b1653ad4f9236a0f41d1d"
server_key_version: "encryptionkey@0"
I20260928 05:15:34.265029 18617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:34.286382 18617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:34.290644 18617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:34.290947 18617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:34.309726 18617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40501
I20260928 05:15:34.309772 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40501 every 8 connection(s)
I20260928 05:15:34.311256 18617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:34.315073 18684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:34.319567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18617
I20260928 05:15:34.319766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:34.320196 388 external_mini_cluster.cc:1428] Setting key 3847462c7ba5f7ae23953125d7370a70
I20260928 05:15:34.326236 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113: Bootstrap starting.
I20260928 05:15:34.329835 18684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:34.331099 18684 log.cc:824] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:34.334576 18684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113: No bootstrap required, opened a new log
Sep 28 05:15:34 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:34.342470 18684 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7834258e4bf4e55b2024ce5a00f3113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40501 } }
I20260928 05:15:34.343012 18684 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:34.343139 18684 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7834258e4bf4e55b2024ce5a00f3113, State: Initialized, Role: FOLLOWER
I20260928 05:15:34.343776 18684 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7834258e4bf4e55b2024ce5a00f3113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40501 } }
I20260928 05:15:34.344071 18684 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:34.344218 18684 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:34.344401 18684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:34.344609 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:34.322156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.0.97.62:40501)
0928 05:15:34.323201 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:34.323216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:34.323257 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:15:34.325103 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:34.325126 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:34.325514 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:34.325842 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:34.327694 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:34.329034 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:34.330948 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:34.331898 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:34.332118 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:34.336904 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:34.336912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:34.336926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:34.336965 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:34.340163 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:34.340943 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:34.340950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:34.340956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:34.341151 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:34.341561 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:34.341565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:34.341567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:34.342074 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:34.342377 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:34.342865 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 05:15:34.343324 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":625,"threads_started":1}
I20260928 05:15:34.346832 18684 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7834258e4bf4e55b2024ce5a00f3113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40501 } }
I20260928 05:15:34.347332 18684 leader_election.cc:302] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7834258e4bf4e55b2024ce5a00f3113; no voters:
I20260928 05:15:34.347980 18684 leader_election.cc:288] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:34.348095 18689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:34.348743 18689 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [term 1 LEADER]: Becoming Leader. State: Replica: e7834258e4bf4e55b2024ce5a00f3113, State: Running, Role: LEADER
I20260928 05:15:34.349347 18689 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7834258e4bf4e55b2024ce5a00f3113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40501 } }
I20260928 05:15:34.350270 18684 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:34.354322 18691 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7834258e4bf4e55b2024ce5a00f3113. Latest consensus state: current_term: 1 leader_uuid: "e7834258e4bf4e55b2024ce5a00f3113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7834258e4bf4e55b2024ce5a00f3113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40501 } } }
I20260928 05:15:34.354265 18690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7834258e4bf4e55b2024ce5a00f3113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7834258e4bf4e55b2024ce5a00f3113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40501 } } }
I20260928 05:15:34.354744 18691 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:34.354753 18690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:34.356853 18699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:34.361835 18699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:34.371145 18699 catalog_manager.cc:1347] Generated new cluster ID: 8822c6b283bf4c2abf98c841a35e5216
I20260928 05:15:34.371281 18699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:34.384624 18699 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:34.385949 18699 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:34.396328 18699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113: Generated new TSK 0
I20260928 05:15:34.397225 18699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:15:34.469547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40501
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:34.801947 18710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:34.802243 18710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:34.802407 18710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:34.809783 18710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:34.809899 18710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:34.810130 18710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:34.820107 18710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:34.821408 18710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:34.822737 18710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:34.836124 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:34.836256 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:34.836149 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:34.837071 18710 server_base.cc:1034] running on GCE node
I20260928 05:15:34.837738 18710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:34.838708 18710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:34.839933 18710 hybrid_clock.cc:648] HybridClock initialized: now 1790572534839891 us; error 52 us; skew 500 ppm
I20260928 05:15:34.840256 18710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:34 dist-test-slave-6718 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:34.843349 18710 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:15:34.844888 18710 webserver.cc:460] Webserver started at http://127.0.97.1:35197/ using document root <none> and password file <none>
I20260928 05:15:34.845587 18710 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:34.845732 18710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:34.846045 18710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:34.848958 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa"
format_stamp: "Formatted at 2026-09-28 05:15:34 on dist-test-slave-6718"
server_key: "3b740e2b0205a26c8d7ee2e04e374961"
server_key_iv: "2dde064bc33fcc687ce20b817c52cc6a"
server_key_version: "encryptionkey@0"
I20260928 05:15:34.849754 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa"
format_stamp: "Formatted at 2026-09-28 05:15:34 on dist-test-slave-6718"
server_key: "3b740e2b0205a26c8d7ee2e04e374961"
server_key_iv: "2dde064bc33fcc687ce20b817c52cc6a"
server_key_version: "encryptionkey@0"
I20260928 05:15:34.855125 18710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260928 05:15:34.858666 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:34.860023 18710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:34.860201 18710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa"
format_stamp: "Formatted at 2026-09-28 05:15:34 on dist-test-slave-6718"
server_key: "3b740e2b0205a26c8d7ee2e04e374961"
server_key_iv: "2dde064bc33fcc687ce20b817c52cc6a"
server_key_version: "encryptionkey@0"
I20260928 05:15:34.860425 18710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:34.875911 18710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:34.880424 18710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:34.880764 18710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:34.881770 18710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:34.883147 18710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:34.883253 18710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:34.883405 18710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:34.883486 18710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:34.916800 18710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42243
I20260928 05:15:34.916889 18837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42243 every 8 connection(s)
I20260928 05:15:34.918057 18710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:34.928081 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18710
I20260928 05:15:34.928269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:15:34.928771 388 external_mini_cluster.cc:1428] Setting key 115e2401282f8846a754c8ca641d634b
Sep 28 05:15:34 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:34.939857 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:34.920803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33527 (local address 127.0.97.62:40501)
0928 05:15:34.921228 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:34.921233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:34.922302 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 05:15:34.924208 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:34.924214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:34.924312 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:34.924442 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:34.925952 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:34.926723 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:34.928663 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:34.929218 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:34.929444 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:34.933589 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:34.933595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:34.933598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:34.933634 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:34.936486 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:34.937311 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:34.937325 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:34.937327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:34.937454 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:34.937917 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:34.937925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:34.937929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:34.938214 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:34.938380 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:34.939374 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 05:15:34.939570 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":201,"threads_started":1}
I20260928 05:15:34.941232 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:34.921371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40501 (local address 127.0.97.1:33527)
0928 05:15:34.922121 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:34.922161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:34.923936 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:34.924684 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:34.924691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:34.925207 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:34.925767 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:34.925785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:34.926866 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:34.926876 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:34.928512 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:34.928529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:34.929475 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:34.929480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:34.929764 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:34.930490 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:34.930511 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:34.933335 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:34.936650 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:34.936661 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:34.936680 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:34.937166 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:34.937573 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:34.937579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:34.937583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:34.937786 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:34.938374 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:34.938382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:34.938826 (+ 444us) client_negotiation.cc:769] Sending connection context
0928 05:15:34.939245 (+ 419us) client_negotiation.cc:240] Negotiation successful
0928 05:15:34.939936 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":301,"threads_started":1}
I20260928 05:15:34.943029 18838 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40501
I20260928 05:15:34.943389 18838 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:34.944164 18838 heartbeater.cc:511] Master 127.0.97.62:40501 requested a full tablet report, sending...
I20260928 05:15:34.946406 18649 ts_manager.cc:194] Registered new tserver with Master: f4bbb61200de4e98aa5fb6e8d782f0fa (127.0.97.1:42243)
I20260928 05:15:34.948066 18649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33527
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:15:34.985446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40501
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:35.313616 18844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:35.313939 18844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:35.314110 18844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:35.321484 18844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:35.321653 18844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:35.321837 18844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:35.331796 18844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:35.333148 18844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:35.334462 18844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:35.347877 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:35.347899 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:35.347934 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:35.348781 18844 server_base.cc:1034] running on GCE node
I20260928 05:15:35.349435 18844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:35.350512 18844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:35.351766 18844 hybrid_clock.cc:648] HybridClock initialized: now 1790572535351717 us; error 38 us; skew 500 ppm
I20260928 05:15:35.352097 18844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:35 dist-test-slave-6718 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:35.355314 18844 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:15:35.356828 18844 webserver.cc:460] Webserver started at http://127.0.97.2:34347/ using document root <none> and password file <none>
I20260928 05:15:35.357542 18844 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:35.357687 18844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:35.358003 18844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:35.361024 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "f9aa8dca9e324a84b33b4594b269a0dd"
format_stamp: "Formatted at 2026-09-28 05:15:35 on dist-test-slave-6718"
server_key: "72c680b1bd1eecea3f7a2d0b6683505d"
server_key_iv: "8bfdebcb9abb1d0561b67c52ab9025d6"
server_key_version: "encryptionkey@0"
I20260928 05:15:35.361819 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f9aa8dca9e324a84b33b4594b269a0dd"
format_stamp: "Formatted at 2026-09-28 05:15:35 on dist-test-slave-6718"
server_key: "72c680b1bd1eecea3f7a2d0b6683505d"
server_key_iv: "8bfdebcb9abb1d0561b67c52ab9025d6"
server_key_version: "encryptionkey@0"
I20260928 05:15:35.367234 18844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 05:15:35.371132 18859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:35.372651 18844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:15:35.372855 18844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "f9aa8dca9e324a84b33b4594b269a0dd"
format_stamp: "Formatted at 2026-09-28 05:15:35 on dist-test-slave-6718"
server_key: "72c680b1bd1eecea3f7a2d0b6683505d"
server_key_iv: "8bfdebcb9abb1d0561b67c52ab9025d6"
server_key_version: "encryptionkey@0"
I20260928 05:15:35.373104 18844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:35.431537 18844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:35.436117 18844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:35.436441 18844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:35.437422 18844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:35.438814 18844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:35.438923 18844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:35.439074 18844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:35.439147 18844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:35.471903 18844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42863
I20260928 05:15:35.471963 18971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42863 every 8 connection(s)
I20260928 05:15:35.473183 18844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:35.480996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18844
I20260928 05:15:35.481199 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:15:35.481633 388 external_mini_cluster.cc:1428] Setting key 58ecaa9b9734c6c0155007214ca97a77
Sep 28 05:15:35 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:35.494011 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:35.475724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40753 (local address 127.0.97.62:40501)
0928 05:15:35.476179 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:35.476184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:35.477080 (+ 896us) server_negotiation.cc:407] Connection header received
0928 05:15:35.478995 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:35.479002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:35.479094 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:35.479212 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:35.480728 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.481545 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.483379 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.483998 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.484140 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:35.488006 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:35.488011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:35.488015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:35.488046 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:35.490728 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:35.491484 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:35.491493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:35.491498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:35.491658 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:35.492067 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:35.492075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:35.492081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:35.492450 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:35.492600 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:35.493258 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 05:15:35.493520 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":246,"threads_started":1}
I20260928 05:15:35.494642 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:35.476159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40501 (local address 127.0.97.2:40753)
0928 05:15:35.476892 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:35.476927 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:35.478728 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:35.479439 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:35.479448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:35.479941 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:35.480532 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:35.480554 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.481699 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.481703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:35.483178 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:35.483190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.484217 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.484222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:35.484429 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:35.485120 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:35.485142 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:35.487774 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:35.490865 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:35.490878 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:35.490898 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:35.491345 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:35.491765 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:35.491768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:35.491770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:35.491951 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:35.492565 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:35.492571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:35.492895 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 05:15:35.493117 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 05:15:35.493506 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":318,"threads_started":1}
I20260928 05:15:35.496225 18972 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40501
I20260928 05:15:35.496567 18972 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:35.497447 18972 heartbeater.cc:511] Master 127.0.97.62:40501 requested a full tablet report, sending...
I20260928 05:15:35.499109 18649 ts_manager.cc:194] Registered new tserver with Master: f9aa8dca9e324a84b33b4594b269a0dd (127.0.97.2:42863)
I20260928 05:15:35.500221 18649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40753
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:15:35.538030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40501
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:35.861382 18978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:35.861666 18978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:35.861843 18978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:35.869629 18978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:35.869755 18978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:35.869938 18978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:35.880250 18978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:35.881646 18978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:35.883021 18978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:35.896226 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:35.896287 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:35.896359 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:35.897315 18978 server_base.cc:1034] running on GCE node
I20260928 05:15:35.897955 18978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:35.898984 18978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:35.900211 18978 hybrid_clock.cc:648] HybridClock initialized: now 1790572535900164 us; error 42 us; skew 500 ppm
I20260928 05:15:35.900566 18978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:35 dist-test-slave-6718 krb5kdc[18604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:35.903725 18978 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:15:35.905283 18978 webserver.cc:460] Webserver started at http://127.0.97.3:41925/ using document root <none> and password file <none>
I20260928 05:15:35.905980 18978 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:35.906126 18978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:35.906445 18978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:35.909363 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "3b08b4c674044a90bae2e3e34c6bbfdd"
format_stamp: "Formatted at 2026-09-28 05:15:35 on dist-test-slave-6718"
server_key: "775314dd870728c7c3f3e1e9e091552f"
server_key_iv: "864e22b378d00afb3675ed039db59aca"
server_key_version: "encryptionkey@0"
I20260928 05:15:35.910178 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3b08b4c674044a90bae2e3e34c6bbfdd"
format_stamp: "Formatted at 2026-09-28 05:15:35 on dist-test-slave-6718"
server_key: "775314dd870728c7c3f3e1e9e091552f"
server_key_iv: "864e22b378d00afb3675ed039db59aca"
server_key_version: "encryptionkey@0"
I20260928 05:15:35.915504 18978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:15:35.919138 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:35.920603 18978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:15:35.920801 18978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "3b08b4c674044a90bae2e3e34c6bbfdd"
format_stamp: "Formatted at 2026-09-28 05:15:35 on dist-test-slave-6718"
server_key: "775314dd870728c7c3f3e1e9e091552f"
server_key_iv: "864e22b378d00afb3675ed039db59aca"
server_key_version: "encryptionkey@0"
I20260928 05:15:35.921041 18978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:35.938791 18978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:35.943506 18978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:35.943825 18978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:35.944810 18978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:35.946247 18978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:35.946353 18978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:35.946511 18978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:35.946604 18978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 05:15:35.951546 18838 heartbeater.cc:503] Master 127.0.97.62:40501 was elected leader, sending a full tablet report...
I20260928 05:15:35.982658 18978 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35681
I20260928 05:15:35.982769 19105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35681 every 8 connection(s)
I20260928 05:15:35.983932 18978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:35.992422 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 18978
I20260928 05:15:35.992620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:15:35.993103 388 external_mini_cluster.cc:1428] Setting key 5d793ef7ad2d02ede9d9cbc3cabb7f05
Sep 28 05:15:35 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:36.005509 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:35.986589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47071 (local address 127.0.97.62:40501)
0928 05:15:35.986807 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:35.986813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:35.988025 (+ 1212us) server_negotiation.cc:407] Connection header received
0928 05:15:35.989963 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:35.989968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:35.990056 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:35.990154 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:35.991802 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.992614 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.994577 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.995279 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.995441 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:35.999383 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:35.999390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:35.999395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:35.999441 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:36.002202 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.003131 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.003137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.003141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.003275 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.003732 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.003737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.003740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.004037 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:36.004201 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.004918 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.005162 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:15:36.006603 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:35.987113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40501 (local address 127.0.97.3:47071)
0928 05:15:35.987849 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:35.987892 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:35.989680 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:35.990400 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:35.990409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:35.991026 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:35.991620 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:35.991639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.992810 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.992814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:35.994337 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:35.994353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:35.995540 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:35.995552 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:35.995813 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:35.996590 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:35.996615 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:35.999132 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:36.002389 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:36.002400 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:36.002419 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:36.002962 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:36.003394 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:36.003399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:36.003403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:36.003608 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:36.004174 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:36.004182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:36.004546 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 05:15:36.004818 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 05:15:36.005391 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":289,"threads_started":1}
I20260928 05:15:36.008365 19106 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40501
I20260928 05:15:36.008705 19106 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:36.009559 19106 heartbeater.cc:511] Master 127.0.97.62:40501 requested a full tablet report, sending...
I20260928 05:15:36.011060 18648 ts_manager.cc:194] Registered new tserver with Master: 3b08b4c674044a90bae2e3e34c6bbfdd (127.0.97.3:35681)
I20260928 05:15:36.011926 18648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47071
I20260928 05:15:36.021760 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:15:36.039145 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.025967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.0.97.62:40501)
0928 05:15:36.026132 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.026137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.026700 (+ 563us) server_negotiation.cc:407] Connection header received
0928 05:15:36.026784 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.026787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.026857 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.026952 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:36.028359 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.029101 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.030426 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.030903 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.031002 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.033547 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:36.033552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:36.033556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:36.033588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:36.036341 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.037174 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.037179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.037181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.037307 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.037733 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.037738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.037742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.038102 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:36.038320 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.038722 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.038863 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:15:36.043308 18648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:15:36.050201 18648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:15:36.075116 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.063766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.0.97.3:35681)
0928 05:15:36.064675 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.064687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.065417 (+ 730us) server_negotiation.cc:407] Connection header received
0928 05:15:36.067031 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.067040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.067280 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.067475 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:36.068466 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.070437 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.072386 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.073865 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.073998 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.074255 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.074575 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.074736 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":603,"threads_started":1}
I20260928 05:15:36.075202 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.065549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42243 (local address 127.0.0.1:54994)
0928 05:15:36.066372 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:36.066390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:36.066482 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:36.067619 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:36.067625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:36.067655 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:36.068187 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.068204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.070070 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.070075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.072441 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.072454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.074244 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.074251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.074411 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.074434 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:15:36.074531 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 05:15:36.074744 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":75,"thread_start_us":581,"threads_started":1}
I20260928 05:15:36.075443 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.065874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.0.97.1:42243)
0928 05:15:36.067044 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.067052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.067076 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:15:36.067167 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.067175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.067389 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.068413 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:36.068595 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.069910 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.072593 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.074017 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.074676 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.074888 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.074954 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.075106 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":829,"threads_started":1}
I20260928 05:15:36.075868 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.063006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42863 (local address 127.0.0.1:40982)
0928 05:15:36.066123 (+ 3117us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:36.066144 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:36.066364 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:36.067208 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:36.067215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:36.067243 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:36.067821 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.067833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.069797 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.069802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.072217 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.072232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.074372 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.074378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.074617 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.074644 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 05:15:36.074720 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 05:15:36.074821 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2975,"mutex_wait_us":28,"thread_start_us":2842,"threads_started":1}
I20260928 05:15:36.076124 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.062989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35681 (local address 127.0.0.1:57840)
0928 05:15:36.065225 (+ 2236us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:36.065276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:36.065522 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:36.067495 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:36.067502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:36.067537 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:36.068252 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.068266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.070607 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.070613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.072227 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.072239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.074051 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.074057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.074270 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.074344 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 05:15:36.074490 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 05:15:36.075733 (+ 1243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1880,"mutex_wait_us":70,"thread_start_us":1702,"threads_started":1}
I20260928 05:15:36.076316 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.063042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.0.97.2:42863)
0928 05:15:36.063754 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.063767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.066246 (+ 2479us) server_negotiation.cc:407] Connection header received
0928 05:15:36.066594 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.066603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.066892 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.067062 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:36.068018 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.069641 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.072432 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.073875 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.074031 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.074225 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.075204 (+ 979us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.075373 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":1046,"threads_started":1}
I20260928 05:15:36.079457 18773 tablet_service.cc:1467] Processing CreateTablet for tablet 01ada0c2dbee45598bc1a968f2273e04 (DEFAULT_TABLE table=test-table [id=9aeb6c2ca68246f98e10744f2d2ac4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:36.079792 19041 tablet_service.cc:1467] Processing CreateTablet for tablet 01ada0c2dbee45598bc1a968f2273e04 (DEFAULT_TABLE table=test-table [id=9aeb6c2ca68246f98e10744f2d2ac4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:36.079792 18907 tablet_service.cc:1467] Processing CreateTablet for tablet 01ada0c2dbee45598bc1a968f2273e04 (DEFAULT_TABLE table=test-table [id=9aeb6c2ca68246f98e10744f2d2ac4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:36.080832 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ada0c2dbee45598bc1a968f2273e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:36.081151 19041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ada0c2dbee45598bc1a968f2273e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:36.081149 18907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ada0c2dbee45598bc1a968f2273e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:36.092931 19124 tablet_bootstrap.cc:492] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Bootstrap starting.
I20260928 05:15:36.095816 19125 tablet_bootstrap.cc:492] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Bootstrap starting.
I20260928 05:15:36.096848 19126 tablet_bootstrap.cc:492] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Bootstrap starting.
I20260928 05:15:36.098464 19124 tablet_bootstrap.cc:654] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:36.099601 19126 tablet_bootstrap.cc:654] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:36.099658 19125 tablet_bootstrap.cc:654] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:36.099773 19124 log.cc:824] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:36.100785 19126 log.cc:824] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:36.100860 19125 log.cc:824] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:36.102949 19124 tablet_bootstrap.cc:492] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: No bootstrap required, opened a new log
I20260928 05:15:36.103236 19124 ts_tablet_manager.cc:1397] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260928 05:15:36.103711 19125 tablet_bootstrap.cc:492] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: No bootstrap required, opened a new log
I20260928 05:15:36.103824 19126 tablet_bootstrap.cc:492] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: No bootstrap required, opened a new log
I20260928 05:15:36.104003 19125 ts_tablet_manager.cc:1397] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 05:15:36.104084 19126 ts_tablet_manager.cc:1397] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 05:15:36.114449 19125 raft_consensus.cc:348] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.115130 19125 raft_consensus.cc:374] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:36.115283 19125 raft_consensus.cc:729] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9aa8dca9e324a84b33b4594b269a0dd, State: Initialized, Role: FOLLOWER
I20260928 05:15:36.115272 19124 raft_consensus.cc:348] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.115957 19124 raft_consensus.cc:374] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:36.116102 19124 raft_consensus.cc:729] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bbb61200de4e98aa5fb6e8d782f0fa, State: Initialized, Role: FOLLOWER
I20260928 05:15:36.116014 19125 consensus_queue.cc:260] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.117309 18972 heartbeater.cc:503] Master 127.0.97.62:40501 was elected leader, sending a full tablet report...
I20260928 05:15:36.117352 19126 raft_consensus.cc:348] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.117780 19126 raft_consensus.cc:374] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:36.117722 19124 consensus_queue.cc:260] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.117897 19126 raft_consensus.cc:729] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b08b4c674044a90bae2e3e34c6bbfdd, State: Initialized, Role: FOLLOWER
I20260928 05:15:36.118117 19125 ts_tablet_manager.cc:1428] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260928 05:15:36.118657 19126 consensus_queue.cc:260] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.119928 19124 ts_tablet_manager.cc:1428] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260928 05:15:36.121016 19106 heartbeater.cc:503] Master 127.0.97.62:40501 was elected leader, sending a full tablet report...
I20260928 05:15:36.122210 19126 ts_tablet_manager.cc:1428] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260928 05:15:36.163218 19131 raft_consensus.cc:482] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:36.163581 19131 raft_consensus.cc:504] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.165510 19131 leader_election.cc:288] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9aa8dca9e324a84b33b4594b269a0dd (127.0.97.2:42863), 3b08b4c674044a90bae2e3e34c6bbfdd (127.0.97.3:35681)
W20260928 05:15:36.170490 18839 tablet.cc:2369] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:36.173383 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.165882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52217 (local address 127.0.97.2:42863)
0928 05:15:36.166169 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.166177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.166931 (+ 754us) server_negotiation.cc:407] Connection header received
0928 05:15:36.167008 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.167013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.167084 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.167197 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:36.168160 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.168925 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.171193 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.172423 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.172563 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.172640 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.172986 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.173124 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260928 05:15:36.173383 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.166475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55677 (local address 127.0.97.3:35681)
0928 05:15:36.166680 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.166685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.166708 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:15:36.166989 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.166994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.167088 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.167188 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:36.168139 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.168914 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.171064 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.172287 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.172413 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.172527 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.172986 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.173134 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:15:36.173659 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.165794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35681 (local address 127.0.97.1:55677)
0928 05:15:36.166594 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:36.166618 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:36.166783 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:36.167223 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:36.167229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:36.167283 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:36.167967 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.167978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.169046 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.169053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.170886 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.170900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.172530 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.172536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.172727 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.172829 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 05:15:36.172932 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 05:15:36.173137 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":162,"spinlock_wait_cycles":29696,"thread_start_us":381,"threads_started":1}
I20260928 05:15:36.173702 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.165976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42863 (local address 127.0.97.1:52217)
0928 05:15:36.166766 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:36.166782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:36.166896 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:36.167223 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:36.167229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:36.167285 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:36.167922 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.167945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.169110 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.169114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.171031 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:36.171049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.172627 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.172633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:36.172796 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.172854 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 05:15:36.172937 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 05:15:36.173137 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":120,"spinlock_wait_cycles":13440,"thread_start_us":581,"threads_started":1}
I20260928 05:15:36.174743 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ada0c2dbee45598bc1a968f2273e04" candidate_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" is_pre_election: true
I20260928 05:15:36.174753 18927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ada0c2dbee45598bc1a968f2273e04" candidate_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" is_pre_election: true
I20260928 05:15:36.175208 18927 raft_consensus.cc:2454] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bbb61200de4e98aa5fb6e8d782f0fa in term 0.
I20260928 05:15:36.175215 19061 raft_consensus.cc:2454] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4bbb61200de4e98aa5fb6e8d782f0fa in term 0.
I20260928 05:15:36.175879 18728 leader_election.cc:302] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b08b4c674044a90bae2e3e34c6bbfdd, f4bbb61200de4e98aa5fb6e8d782f0fa, f9aa8dca9e324a84b33b4594b269a0dd; no voters:
I20260928 05:15:36.176294 19131 raft_consensus.cc:2781] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:15:36.176451 19131 raft_consensus.cc:482] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:15:36.176570 19131 raft_consensus.cc:3037] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:36.178824 19131 raft_consensus.cc:504] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.179432 19131 leader_election.cc:288] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [CANDIDATE]: Term 1 election: Requested vote from peers f9aa8dca9e324a84b33b4594b269a0dd (127.0.97.2:42863), 3b08b4c674044a90bae2e3e34c6bbfdd (127.0.97.3:35681)
I20260928 05:15:36.179808 18927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ada0c2dbee45598bc1a968f2273e04" candidate_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9aa8dca9e324a84b33b4594b269a0dd"
I20260928 05:15:36.179914 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ada0c2dbee45598bc1a968f2273e04" candidate_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd"
I20260928 05:15:36.180039 18927 raft_consensus.cc:3037] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:36.180097 19061 raft_consensus.cc:3037] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:36.183281 18927 raft_consensus.cc:2454] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bbb61200de4e98aa5fb6e8d782f0fa in term 1.
I20260928 05:15:36.183281 19061 raft_consensus.cc:2454] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4bbb61200de4e98aa5fb6e8d782f0fa in term 1.
I20260928 05:15:36.183671 18728 leader_election.cc:302] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4bbb61200de4e98aa5fb6e8d782f0fa, f9aa8dca9e324a84b33b4594b269a0dd; no voters:
I20260928 05:15:36.183971 19131 raft_consensus.cc:2781] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:36.184373 19131 raft_consensus.cc:686] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 1 LEADER]: Becoming Leader. State: Replica: f4bbb61200de4e98aa5fb6e8d782f0fa, State: Running, Role: LEADER
I20260928 05:15:36.184875 19131 consensus_queue.cc:237] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } }
I20260928 05:15:36.189851 18647 catalog_manager.cc:5495] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa reported cstate change: term changed from 0 to 1, leader changed from <none> to f4bbb61200de4e98aa5fb6e8d782f0fa (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bbb61200de4e98aa5fb6e8d782f0fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42243 } health_report { overall_health: HEALTHY } } }
W20260928 05:15:36.225286 18973 tablet.cc:2369] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:36.232862 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.227411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.0.97.1:42243)
0928 05:15:36.227624 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.227629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.227646 (+ 17us) server_negotiation.cc:407] Connection header received
0928 05:15:36.227787 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.227790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.227867 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.227995 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:15:36.228435 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.229148 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.230625 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.231037 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.231135 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.231723 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:15:36.232210 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:15:36.232378 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.232520 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.232587 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 05:15:36.235479 19107 tablet.cc:2369] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:36.241005 18927 raft_consensus.cc:1261] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 1 FOLLOWER]: Refusing update from remote peer f4bbb61200de4e98aa5fb6e8d782f0fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:36.241004 19061 raft_consensus.cc:1261] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 1 FOLLOWER]: Refusing update from remote peer f4bbb61200de4e98aa5fb6e8d782f0fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:36.242054 19135 consensus_queue.cc:1035] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b08b4c674044a90bae2e3e34c6bbfdd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:36.242494 19131 consensus_queue.cc:1035] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9aa8dca9e324a84b33b4594b269a0dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:36.253333 19138 mvcc.cc:205] Tried to move back new op lower bound from 7334185108432498688 to 7334185108216860672. Current Snapshot: MvccSnapshot[applied={T|T < 7334185108432498688}]
I20260928 05:15:36.258014 19139 mvcc.cc:205] Tried to move back new op lower bound from 7334185108432498688 to 7334185108216860672. Current Snapshot: MvccSnapshot[applied={T|T < 7334185108432498688}]
I20260928 05:15:36.268549 18647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:36.268828 18647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:36.272187 18647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7834258e4bf4e55b2024ce5a00f3113: Sending DeleteTablet for 3 replicas of tablet 01ada0c2dbee45598bc1a968f2273e04
I20260928 05:15:36.273403 18907 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ada0c2dbee45598bc1a968f2273e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40982
I20260928 05:15:36.273571 19041 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ada0c2dbee45598bc1a968f2273e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260928 05:15:36.274276 18773 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ada0c2dbee45598bc1a968f2273e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54994
I20260928 05:15:36.275652 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18710
I20260928 05:15:36.284878 19147 tablet_replica.cc:330] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: stopping tablet replica
I20260928 05:15:36.286017 19147 raft_consensus.cc:2229] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:36.287237 19147 raft_consensus.cc:2258] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:36.291424 19147 ts_tablet_manager.cc:1905] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:36.295384 19149 tablet_replica.cc:330] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: stopping tablet replica
I20260928 05:15:36.296653 19149 raft_consensus.cc:2229] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:15:36.297309 19148 tablet_replica.cc:330] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: stopping tablet replica
I20260928 05:15:36.298146 19149 raft_consensus.cc:2258] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:36.298713 19148 raft_consensus.cc:2229] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:36.299445 19148 raft_consensus.cc:2258] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:36.301959 19148 ts_tablet_manager.cc:1905] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:36.301972 19149 ts_tablet_manager.cc:1905] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 05:15:36 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:15:36.309396 19149 ts_tablet_manager.cc:1918] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:36.311712 19147 ts_tablet_manager.cc:1918] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:36.311971 19147 log.cc:1196] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/01ada0c2dbee45598bc1a968f2273e04
I20260928 05:15:36.312593 19147 ts_tablet_manager.cc:1939] T 01ada0c2dbee45598bc1a968f2273e04 P f9aa8dca9e324a84b33b4594b269a0dd: Deleting consensus metadata
I20260928 05:15:36.312813 19149 log.cc:1196] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/01ada0c2dbee45598bc1a968f2273e04
I20260928 05:15:36.313412 19149 ts_tablet_manager.cc:1939] T 01ada0c2dbee45598bc1a968f2273e04 P f4bbb61200de4e98aa5fb6e8d782f0fa: Deleting consensus metadata
I20260928 05:15:36.315814 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.277150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.0.97.1:42243)
0928 05:15:36.277331 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.277339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.296035 (+ 18696us) server_negotiation.cc:407] Connection header received
0928 05:15:36.296260 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.296266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.296360 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.296459 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:36.298381 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.299750 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.301710 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.302400 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.302513 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.308105 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:36.308123 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:36.308147 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:36.308202 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:36.312381 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.313253 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.313263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.313267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.313450 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.313879 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.313884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.313888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.314307 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:36.314608 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.315190 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.315448 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:15:36.316672 18633 catalog_manager.cc:4841] TS f4bbb61200de4e98aa5fb6e8d782f0fa (127.0.97.1:42243): tablet 01ada0c2dbee45598bc1a968f2273e04 (table test-table [id=9aeb6c2ca68246f98e10744f2d2ac4cf]) successfully deleted
I20260928 05:15:36.317013 18635 catalog_manager.cc:4841] TS f9aa8dca9e324a84b33b4594b269a0dd (127.0.97.2:42863): tablet 01ada0c2dbee45598bc1a968f2273e04 (table test-table [id=9aeb6c2ca68246f98e10744f2d2ac4cf]) successfully deleted
I20260928 05:15:36.318107 18833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:36.319371 19148 ts_tablet_manager.cc:1918] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:36.319638 19148 log.cc:1196] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/01ada0c2dbee45598bc1a968f2273e04
I20260928 05:15:36.320286 19148 ts_tablet_manager.cc:1939] T 01ada0c2dbee45598bc1a968f2273e04 P 3b08b4c674044a90bae2e3e34c6bbfdd: Deleting consensus metadata
I20260928 05:15:36.322016 18633 catalog_manager.cc:4841] TS 3b08b4c674044a90bae2e3e34c6bbfdd (127.0.97.3:35681): tablet 01ada0c2dbee45598bc1a968f2273e04 (table test-table [id=9aeb6c2ca68246f98e10744f2d2ac4cf]) successfully deleted
I20260928 05:15:36.420944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18710
I20260928 05:15:36.439507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18844
Sep 28 05:15:36 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:15:36.458175 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.440304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.97.2:42863)
0928 05:15:36.440511 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.440515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.440535 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:15:36.440771 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.440776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.440850 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.440978 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:36.442272 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.443100 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.444493 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.444936 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.445047 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.451225 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:36.451246 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:36.451290 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:36.451346 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:36.455142 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.456027 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.456037 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.456041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.456211 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.456651 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.456656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.456658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.457071 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:36.457320 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.457704 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.457886 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 05:15:36.458913 18967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:36.548306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18844
I20260928 05:15:36.565258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18978
Sep 28 05:15:36 dist-test-slave-6718 krb5kdc[18604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:15:36.580411 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:36.566003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.0.97.3:35681)
0928 05:15:36.566202 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:36.566207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:36.566226 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:15:36.566284 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:36.566287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:36.566371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:36.566456 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:36.567786 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.568643 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.570016 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:36.570415 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:36.570513 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:36.574995 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:36.575016 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:36.575032 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:36.575073 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:36.577880 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.578504 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.578511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.578513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.578707 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:36.579029 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:36.579032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:36.579034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:36.579353 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:36.579572 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:36.579846 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 05:15:36.580016 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 05:15:36.581086 19101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:36.671854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18978
I20260928 05:15:36.688706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 18617
I20260928 05:15:36.689817 18679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:36.801188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 18617
2026-09-28T05:15:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4306 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T05:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:36Z Disabled control of system clock
I20260928 05:15:36.877478 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45659
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:15:37.222324 19158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:37.222657 19158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:37.222740 19158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:37.230468 19158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:37.230641 19158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:37.230697 19158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:37.230748 19158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:37.230791 19158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:37.239992 19158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:37.241247 19158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:37.243085 19158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:37.255172 19166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:37.255195 19163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:37.255098 19164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:37.256219 19158 server_base.cc:1034] running on GCE node
I20260928 05:15:37.257071 19158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:37.258500 19158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:37.259788 19158 hybrid_clock.cc:648] HybridClock initialized: now 1790572537259719 us; error 61 us; skew 500 ppm
I20260928 05:15:37.260166 19158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:37.262393 19158 webserver.cc:460] Webserver started at http://127.0.97.62:41127/ using document root <none> and password file <none>
I20260928 05:15:37.263147 19158 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:37.263281 19158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:37.263589 19158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:37.266491 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f"
format_stamp: "Formatted at 2026-09-28 05:15:37 on dist-test-slave-6718"
server_key: "b2d472c8e8116d15cac9088f72ce22f2"
server_key_iv: "a051679adbdbfa778865edd1cb9277d6"
server_key_version: "encryptionkey@0"
I20260928 05:15:37.267349 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f"
format_stamp: "Formatted at 2026-09-28 05:15:37 on dist-test-slave-6718"
server_key: "b2d472c8e8116d15cac9088f72ce22f2"
server_key_iv: "a051679adbdbfa778865edd1cb9277d6"
server_key_version: "encryptionkey@0"
I20260928 05:15:37.272289 19158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:15:37.275688 19172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:37.276925 19158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:37.277105 19158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f"
format_stamp: "Formatted at 2026-09-28 05:15:37 on dist-test-slave-6718"
server_key: "b2d472c8e8116d15cac9088f72ce22f2"
server_key_iv: "a051679adbdbfa778865edd1cb9277d6"
server_key_version: "encryptionkey@0"
I20260928 05:15:37.277379 19158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:37.290409 19158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:37.291232 19158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:37.291711 19158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:37.311370 19158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45659
I20260928 05:15:37.311393 19223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45659 every 8 connection(s)
I20260928 05:15:37.312770 19158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:37.316761 19224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:37.320896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19158
I20260928 05:15:37.321063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:37.321511 388 external_mini_cluster.cc:1428] Setting key 98fe58e2c23b473fe0e322a558e408d8
I20260928 05:15:37.329226 19224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: Bootstrap starting.
I20260928 05:15:37.332813 19224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:37.334141 19224 log.cc:824] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:37.337575 19224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: No bootstrap required, opened a new log
I20260928 05:15:37.337802 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:37.323265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.0.97.62:45659)
0928 05:15:37.324316 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:37.324332 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:37.324382 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:15:37.327387 (+ 3005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:37.327414 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:37.328004 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:37.328430 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:37.328963 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:37.330968 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:37.332291 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:37.333388 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:37.333610 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:37.335102 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:37.335113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:37.335127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:37.335776 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:37.335879 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:37.335906 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:37.336027 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:37.336263 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 05:15:37.336589 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":330,"threads_started":1}
I20260928 05:15:37.345329 19224 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260928 05:15:37.345775 19224 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:37.345920 19224 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fd5491fe0b4ebeb0b571282ed89a7f, State: Initialized, Role: FOLLOWER
I20260928 05:15:37.346460 19224 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260928 05:15:37.346766 19224 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:37.346920 19224 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:37.347095 19224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:37.349184 19224 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260928 05:15:37.349673 19224 leader_election.cc:302] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0fd5491fe0b4ebeb0b571282ed89a7f; no voters:
I20260928 05:15:37.350396 19224 leader_election.cc:288] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:37.350621 19229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:37.351450 19229 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [term 1 LEADER]: Becoming Leader. State: Replica: d0fd5491fe0b4ebeb0b571282ed89a7f, State: Running, Role: LEADER
I20260928 05:15:37.351998 19229 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } }
I20260928 05:15:37.352494 19224 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:37.354561 19231 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0fd5491fe0b4ebeb0b571282ed89a7f. Latest consensus state: current_term: 1 leader_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } } }
I20260928 05:15:37.354938 19231 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:37.355507 19230 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fd5491fe0b4ebeb0b571282ed89a7f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45659 } } }
I20260928 05:15:37.355792 19230 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f [sys.catalog]: This master's current role is: LEADER
W20260928 05:15:37.364394 19246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:15:37.364593 19246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:15:37.365780 19237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:37.368753 19237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:37.379972 19237 catalog_manager.cc:1347] Generated new cluster ID: be6660a950804951beb469ada15a945d
I20260928 05:15:37.380187 19237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:37.390376 19237 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:37.392028 19237 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:37.399420 19237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: Generated new TSK 0
I20260928 05:15:37.400240 19237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:37.406841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:15:37.731264 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:37.731542 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:37.731699 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:37.738955 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:37.739049 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:37.739229 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:37.748425 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:37.749737 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:37.751483 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:37.765215 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:37.765269 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:37.765239 19257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:37.765966 19249 server_base.cc:1034] running on GCE node
I20260928 05:15:37.766669 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:37.767650 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:37.768884 19249 hybrid_clock.cc:648] HybridClock initialized: now 1790572537768845 us; error 48 us; skew 500 ppm
I20260928 05:15:37.769212 19249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:37.771571 19249 webserver.cc:460] Webserver started at http://127.0.97.1:34591/ using document root <none> and password file <none>
I20260928 05:15:37.772256 19249 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:37.772387 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:37.772708 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:37.775681 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "ae87126d87c14c1381593b355746c1a2"
format_stamp: "Formatted at 2026-09-28 05:15:37 on dist-test-slave-6718"
server_key: "a870b0666f0388c851654457912c5ada"
server_key_iv: "e92763250c6bbc005fdbd95429a62c7f"
server_key_version: "encryptionkey@0"
I20260928 05:15:37.776474 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ae87126d87c14c1381593b355746c1a2"
format_stamp: "Formatted at 2026-09-28 05:15:37 on dist-test-slave-6718"
server_key: "a870b0666f0388c851654457912c5ada"
server_key_iv: "e92763250c6bbc005fdbd95429a62c7f"
server_key_version: "encryptionkey@0"
I20260928 05:15:37.781469 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 05:15:37.784889 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:37.786211 19249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:15:37.786419 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "ae87126d87c14c1381593b355746c1a2"
format_stamp: "Formatted at 2026-09-28 05:15:37 on dist-test-slave-6718"
server_key: "a870b0666f0388c851654457912c5ada"
server_key_iv: "e92763250c6bbc005fdbd95429a62c7f"
server_key_version: "encryptionkey@0"
I20260928 05:15:37.786716 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:37.812455 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:37.813297 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:37.813540 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:37.814450 19249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:37.815930 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:37.816016 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:37.816095 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:37.816152 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:37.849138 19249 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36617
I20260928 05:15:37.849231 19375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36617 every 8 connection(s)
I20260928 05:15:37.850512 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:37.850899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19249
I20260928 05:15:37.851083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:15:37.851683 388 external_mini_cluster.cc:1428] Setting key 825a9a4c4529a2e27b4f6e7dbb0670f0
I20260928 05:15:37.854941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:15:37.864713 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:37.853729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44203 (local address 127.0.97.62:45659)
0928 05:15:37.854248 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:37.854256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:37.855687 (+ 1431us) server_negotiation.cc:407] Connection header received
0928 05:15:37.857673 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:37.857679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:37.857764 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:37.857892 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:37.859038 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:37.859780 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:37.861346 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:37.861943 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:37.862088 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:37.863852 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:37.863858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:37.863861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:37.863879 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:37.863926 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:37.863938 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:37.864035 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:37.864267 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 05:15:37.864363 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":231,"threads_started":1}
I20260928 05:15:37.865739 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:37.854391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45659 (local address 127.0.97.1:44203)
0928 05:15:37.855506 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:37.855557 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:37.857405 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:37.858123 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:37.858131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:37.858230 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:37.858876 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:37.858893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:37.859963 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:37.859968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:37.861194 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:37.861205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:37.862243 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:37.862278 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:37.862672 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:37.863510 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:37.863513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:37.863552 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:37.863638 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:37.864034 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:37.864048 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:37.864058 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:15:37.864244 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 05:15:37.864680 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":397,"threads_started":1}
I20260928 05:15:37.867345 19376 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45659
I20260928 05:15:37.867691 19376 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:37.868623 19376 heartbeater.cc:511] Master 127.0.97.62:45659 requested a full tablet report, sending...
I20260928 05:15:37.871096 19189 ts_manager.cc:194] Registered new tserver with Master: ae87126d87c14c1381593b355746c1a2 (127.0.97.1:36617)
I20260928 05:15:37.873437 19189 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:44203
W20260928 05:15:38.189158 19380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:38.189450 19380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:38.189538 19380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:38.196858 19380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:38.196965 19380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:38.197073 19380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:38.206848 19380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:38.208091 19380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:38.209781 19380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:38.223124 19386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:38.223097 19388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:38.223196 19385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:38.223871 19380 server_base.cc:1034] running on GCE node
I20260928 05:15:38.224548 19380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:38.225463 19380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:38.226624 19380 hybrid_clock.cc:648] HybridClock initialized: now 1790572538226582 us; error 35 us; skew 500 ppm
I20260928 05:15:38.227028 19380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:38.229447 19380 webserver.cc:460] Webserver started at http://127.0.97.2:37225/ using document root <none> and password file <none>
I20260928 05:15:38.230159 19380 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:38.230274 19380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:38.230759 19380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:38.233596 19380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "af3495dc171a4370baa4a67eda68a47c"
format_stamp: "Formatted at 2026-09-28 05:15:38 on dist-test-slave-6718"
server_key: "e2edd29e07cd15792998053cf4ea3277"
server_key_iv: "50085d025b0547ad752f94a08f841c8d"
server_key_version: "encryptionkey@0"
I20260928 05:15:38.234360 19380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "af3495dc171a4370baa4a67eda68a47c"
format_stamp: "Formatted at 2026-09-28 05:15:38 on dist-test-slave-6718"
server_key: "e2edd29e07cd15792998053cf4ea3277"
server_key_iv: "50085d025b0547ad752f94a08f841c8d"
server_key_version: "encryptionkey@0"
I20260928 05:15:38.239418 19380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 05:15:38.242865 19394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:38.244406 19380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:15:38.244587 19380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "af3495dc171a4370baa4a67eda68a47c"
format_stamp: "Formatted at 2026-09-28 05:15:38 on dist-test-slave-6718"
server_key: "e2edd29e07cd15792998053cf4ea3277"
server_key_iv: "50085d025b0547ad752f94a08f841c8d"
server_key_version: "encryptionkey@0"
I20260928 05:15:38.244791 19380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:38.263130 19380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:38.263943 19380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:38.264220 19380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:38.265199 19380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:38.266721 19380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:38.266809 19380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:38.266952 19380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:38.267046 19380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:38.300153 19380 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44439
I20260928 05:15:38.300213 19506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44439 every 8 connection(s)
I20260928 05:15:38.302016 19380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:38.312096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19380
I20260928 05:15:38.312275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:15:38.312798 388 external_mini_cluster.cc:1428] Setting key c8c7f8b42de73f5303b22f16dec0185d
I20260928 05:15:38.315997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--builtin_ntp_servers=127.0.97.20:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:15:38.317190 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.305071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45863 (local address 127.0.97.62:45659)
0928 05:15:38.305257 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:38.305261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:38.306694 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 05:15:38.308530 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:38.308536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:38.308632 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:38.308741 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:38.309879 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.310653 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.312617 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.313299 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.313431 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.316331 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:38.316337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:38.316342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:38.316364 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:38.316405 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:38.316419 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:38.316513 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:38.316786 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 05:15:38.316887 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:15:38.318686 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.305574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45659 (local address 127.0.97.2:45863)
0928 05:15:38.306478 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:38.306517 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:38.308220 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:38.308984 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:38.308992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:38.309089 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:38.309695 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.309711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.310822 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.310828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.312442 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.312458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.313554 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.313563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.313857 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.314673 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:38.314679 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:38.314736 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:38.314851 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:38.316540 (+ 1689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:38.316553 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:38.316565 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 05:15:38.316771 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 05:15:38.317241 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":283,"threads_started":1}
I20260928 05:15:38.320307 19507 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45659
I20260928 05:15:38.320645 19507 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:38.321563 19507 heartbeater.cc:511] Master 127.0.97.62:45659 requested a full tablet report, sending...
I20260928 05:15:38.323173 19189 ts_manager.cc:194] Registered new tserver with Master: af3495dc171a4370baa4a67eda68a47c (127.0.97.2:44439)
I20260928 05:15:38.324445 19189 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45863
W20260928 05:15:38.647755 19510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:38.648046 19510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:38.648180 19510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:38.655716 19510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:38.655814 19510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:38.655980 19510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:38.665226 19510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:38.666585 19510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:38.668324 19510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:38.681463 19516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:38.681430 19518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:38.681510 19515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:38.683902 19510 server_base.cc:1034] running on GCE node
I20260928 05:15:38.684501 19510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:38.685391 19510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:38.686630 19510 hybrid_clock.cc:648] HybridClock initialized: now 1790572538686564 us; error 59 us; skew 500 ppm
I20260928 05:15:38.686949 19510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:38.689248 19510 webserver.cc:460] Webserver started at http://127.0.97.3:36449/ using document root <none> and password file <none>
I20260928 05:15:38.689934 19510 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:38.690047 19510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:38.690438 19510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:38.693331 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "edd85b5f98784d1d9013c7efece08f14"
format_stamp: "Formatted at 2026-09-28 05:15:38 on dist-test-slave-6718"
server_key: "f700df2c8fb3db877007c70eaf7d239b"
server_key_iv: "9824cc5cfb3d6ecc581c4e700d481049"
server_key_version: "encryptionkey@0"
I20260928 05:15:38.694108 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "edd85b5f98784d1d9013c7efece08f14"
format_stamp: "Formatted at 2026-09-28 05:15:38 on dist-test-slave-6718"
server_key: "f700df2c8fb3db877007c70eaf7d239b"
server_key_iv: "9824cc5cfb3d6ecc581c4e700d481049"
server_key_version: "encryptionkey@0"
I20260928 05:15:38.699060 19510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:15:38.702397 19524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:38.703682 19510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:38.703846 19510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "edd85b5f98784d1d9013c7efece08f14"
format_stamp: "Formatted at 2026-09-28 05:15:38 on dist-test-slave-6718"
server_key: "f700df2c8fb3db877007c70eaf7d239b"
server_key_iv: "9824cc5cfb3d6ecc581c4e700d481049"
server_key_version: "encryptionkey@0"
I20260928 05:15:38.704077 19510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:38.720515 19510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:38.721405 19510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:38.721693 19510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:38.722823 19510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:38.724538 19510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:38.724630 19510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:38.724726 19510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:38.724807 19510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:38.757997 19510 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44863
I20260928 05:15:38.758100 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44863 every 8 connection(s)
I20260928 05:15:38.759455 19510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:38.765719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19510
I20260928 05:15:38.765890 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:15:38.766363 388 external_mini_cluster.cc:1428] Setting key dd2af506a599f1ad5a2ded24855709b1
I20260928 05:15:38.772753 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.762483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58607 (local address 127.0.97.62:45659)
0928 05:15:38.762719 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:38.762725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:38.764219 (+ 1494us) server_negotiation.cc:407] Connection header received
0928 05:15:38.766087 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:38.766094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:38.766189 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:38.766299 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:38.767553 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.768350 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.769956 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.770369 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.770526 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.771943 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:38.771948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:38.771951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:38.771970 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:38.772008 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:38.772019 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:38.772115 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:38.772343 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 05:15:38.772442 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 05:15:38.773813 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.763118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45659 (local address 127.0.97.3:58607)
0928 05:15:38.764054 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:38.764091 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:38.765724 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:38.766622 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:38.766631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:38.766724 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:38.767364 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.767390 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.768516 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.768521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.769806 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.769817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.770640 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.770648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.770880 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.771613 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:38.771615 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:38.771656 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:38.771739 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:38.772114 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:38.772128 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:38.772138 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 05:15:38.772317 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 05:15:38.772743 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":307,"threads_started":1}
I20260928 05:15:38.775133 19637 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45659
I20260928 05:15:38.775421 19637 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:38.776154 19637 heartbeater.cc:511] Master 127.0.97.62:45659 requested a full tablet report, sending...
I20260928 05:15:38.777603 19189 ts_manager.cc:194] Registered new tserver with Master: edd85b5f98784d1d9013c7efece08f14 (127.0.97.3:44863)
I20260928 05:15:38.778348 19189 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:58607
I20260928 05:15:38.781474 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:15:38.791123 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.785379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.0.97.62:45659)
0928 05:15:38.785576 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:38.785581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:38.785987 (+ 406us) server_negotiation.cc:407] Connection header received
0928 05:15:38.786112 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:38.786115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:38.786182 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:38.786259 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:38.786739 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.787412 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.788750 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.789117 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.789202 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.790305 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:38.790311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:38.790314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:38.790331 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:38.790366 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:38.790377 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:38.790499 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:38.790782 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 05:15:38.790839 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:15:38.795195 19189 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:15:38.802877 19189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:15:38.839406 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.816006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44439 (local address 127.0.0.1:42674)
0928 05:15:38.823223 (+ 7217us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:38.823275 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:38.823448 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:38.831349 (+ 7901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:38.831360 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:38.831398 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:38.831899 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.831918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.834280 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.834285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.836629 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.836641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.838214 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.838221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.838509 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.838609 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 05:15:38.838751 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 05:15:38.838956 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6953,"mutex_wait_us":47,"thread_start_us":6771,"threads_started":1}
I20260928 05:15:38.840337 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.816001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36617 (local address 127.0.0.1:58180)
0928 05:15:38.829160 (+ 13159us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:38.829190 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:38.829377 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:38.832235 (+ 2858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:38.832240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:38.832268 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:38.832680 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.832693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.834747 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.834753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.837254 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.837269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.839416 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.839425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.839711 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.839742 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:15:38.839836 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 05:15:38.839985 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12947,"mutex_wait_us":326,"thread_start_us":12393,"threads_started":1}
I20260928 05:15:38.840489 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.816122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.97.1:36617)
0928 05:15:38.831360 (+ 15238us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:38.831369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:38.831398 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:15:38.831509 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:38.831520 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:38.831783 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:38.832890 (+ 1107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:38.833117 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.834614 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.837477 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.838911 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.839052 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.839272 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:38.839891 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 05:15:38.840099 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15067,"thread_start_us":15845,"threads_started":1}
I20260928 05:15:38.841176 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.823932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44863 (local address 127.0.0.1:60972)
0928 05:15:38.831016 (+ 7084us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:38.831042 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:38.831189 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:38.833717 (+ 2528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:38.833721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:38.833744 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:38.834063 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.834072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.836250 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.836256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.838266 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:38.838283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.840376 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.840383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:38.840582 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.840623 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 05:15:38.840731 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 05:15:38.840846 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6912,"mutex_wait_us":1,"thread_start_us":7353,"threads_started":1}
I20260928 05:15:38.841324 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.824030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.0.97.3:44863)
0928 05:15:38.833289 (+ 9259us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:38.833294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:38.833319 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:15:38.833387 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:38.833392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:38.833544 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:38.833730 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:38.834259 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.836104 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.838484 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.839860 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.839973 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.840219 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:38.840781 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 05:15:38.840940 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9108,"thread_start_us":8964,"threads_started":1}
I20260928 05:15:38.841590 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:38.816824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.97.2:44439)
0928 05:15:38.830101 (+ 13277us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:38.830117 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:38.830162 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:15:38.830310 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:38.830325 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:38.830782 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:38.832131 (+ 1349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:38.832353 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.834094 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.836799 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:38.837995 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:38.838124 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:38.838368 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:38.840770 (+ 2402us) server_negotiation.cc:299] Negotiation successful
0928 05:15:38.840954 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13000,"thread_start_us":14158,"threads_started":1}
I20260928 05:15:38.844684 19572 tablet_service.cc:1467] Processing CreateTablet for tablet 8e09cf667a4d46e6b224fe5c71a30f1c (DEFAULT_TABLE table=test-table [id=1e731546094c4b5d851dcc5738b26228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:38.844791 19311 tablet_service.cc:1467] Processing CreateTablet for tablet 8e09cf667a4d46e6b224fe5c71a30f1c (DEFAULT_TABLE table=test-table [id=1e731546094c4b5d851dcc5738b26228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:38.845072 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 8e09cf667a4d46e6b224fe5c71a30f1c (DEFAULT_TABLE table=test-table [id=1e731546094c4b5d851dcc5738b26228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:38.846156 19572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e09cf667a4d46e6b224fe5c71a30f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:38.846477 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e09cf667a4d46e6b224fe5c71a30f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:38.846650 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e09cf667a4d46e6b224fe5c71a30f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:38.865062 19656 tablet_bootstrap.cc:492] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Bootstrap starting.
I20260928 05:15:38.867306 19657 tablet_bootstrap.cc:492] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Bootstrap starting.
I20260928 05:15:38.869351 19655 tablet_bootstrap.cc:492] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Bootstrap starting.
I20260928 05:15:38.870721 19656 tablet_bootstrap.cc:654] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:38.871276 19657 tablet_bootstrap.cc:654] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:38.871830 19656 log.cc:824] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:38.872622 19657 log.cc:824] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:38.873729 19655 tablet_bootstrap.cc:654] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:38.875097 19655 log.cc:824] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:38.877368 19657 tablet_bootstrap.cc:492] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: No bootstrap required, opened a new log
I20260928 05:15:38.877686 19657 ts_tablet_manager.cc:1397] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260928 05:15:38.877873 19656 tablet_bootstrap.cc:492] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: No bootstrap required, opened a new log
I20260928 05:15:38.877887 19376 heartbeater.cc:503] Master 127.0.97.62:45659 was elected leader, sending a full tablet report...
I20260928 05:15:38.878209 19656 ts_tablet_manager.cc:1397] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260928 05:15:38.878831 19655 tablet_bootstrap.cc:492] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: No bootstrap required, opened a new log
I20260928 05:15:38.879149 19655 ts_tablet_manager.cc:1397] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 05:15:38.887970 19656 raft_consensus.cc:348] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:38.888442 19656 raft_consensus.cc:374] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:38.888566 19656 raft_consensus.cc:729] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae87126d87c14c1381593b355746c1a2, State: Initialized, Role: FOLLOWER
I20260928 05:15:38.888705 19657 raft_consensus.cc:348] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:38.889155 19657 raft_consensus.cc:374] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:38.889261 19657 raft_consensus.cc:729] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3495dc171a4370baa4a67eda68a47c, State: Initialized, Role: FOLLOWER
I20260928 05:15:38.889382 19656 consensus_queue.cc:260] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:38.889990 19657 consensus_queue.cc:260] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:38.890695 19655 raft_consensus.cc:348] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:38.891294 19655 raft_consensus.cc:374] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:38.891450 19655 raft_consensus.cc:729] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd85b5f98784d1d9013c7efece08f14, State: Initialized, Role: FOLLOWER
I20260928 05:15:38.891636 19656 ts_tablet_manager.cc:1428] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Time spent starting tablet: real 0.013s user 0.005s sys 0.010s
I20260928 05:15:38.892298 19655 consensus_queue.cc:260] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:38.893323 19507 heartbeater.cc:503] Master 127.0.97.62:45659 was elected leader, sending a full tablet report...
I20260928 05:15:38.894168 19657 ts_tablet_manager.cc:1428] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260928 05:15:38.895116 19637 heartbeater.cc:503] Master 127.0.97.62:45659 was elected leader, sending a full tablet report...
I20260928 05:15:38.895715 19655 ts_tablet_manager.cc:1428] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260928 05:15:39.011431 19638 tablet.cc:2369] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:39.018980 19661 raft_consensus.cc:482] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:39.019408 19661 raft_consensus.cc:504] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:39.021394 19661 leader_election.cc:288] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3495dc171a4370baa4a67eda68a47c (127.0.97.2:44439), edd85b5f98784d1d9013c7efece08f14 (127.0.97.3:44863)
I20260928 05:15:39.029440 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.021779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41685 (local address 127.0.97.2:44439)
0928 05:15:39.021981 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:39.021985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:39.022498 (+ 513us) server_negotiation.cc:407] Connection header received
0928 05:15:39.022793 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:39.022799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:39.022917 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:39.023022 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:39.023818 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.024655 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.027176 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.028512 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.028649 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.028790 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:39.029105 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 05:15:39.029226 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:15:39.029596 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.021609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44439 (local address 127.0.97.1:41685)
0928 05:15:39.022360 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:39.022387 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:39.022581 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:39.023110 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:39.023114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:39.023142 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:39.023584 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:39.023598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.024794 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.024798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:39.026957 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:39.026968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.028722 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.028727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:39.028914 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.028940 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 05:15:39.029065 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 05:15:39.029219 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":28,"thread_start_us":382,"threads_started":1}
I20260928 05:15:39.030824 19462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e09cf667a4d46e6b224fe5c71a30f1c" candidate_uuid: "ae87126d87c14c1381593b355746c1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3495dc171a4370baa4a67eda68a47c" is_pre_election: true
I20260928 05:15:39.031229 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.022776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56919 (local address 127.0.97.3:44863)
0928 05:15:39.022994 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:39.023002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:39.023326 (+ 324us) server_negotiation.cc:407] Connection header received
0928 05:15:39.023530 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:39.023538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:39.023613 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:39.023699 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:39.025448 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.026520 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.029106 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.030323 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.030467 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.030588 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:39.030874 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 05:15:39.031006 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 05:15:39.031315 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.022463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44863 (local address 127.0.97.1:56919)
0928 05:15:39.023225 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:39.023240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:39.023398 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:39.024817 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:39.024821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:39.024847 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:39.025257 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:39.025267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.026671 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.026676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:39.028902 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:39.028914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.030510 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.030517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:39.030701 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.030724 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:15:39.030834 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 05:15:39.030968 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":5,"spinlock_wait_cycles":6528,"thread_start_us":383,"threads_started":1}
I20260928 05:15:39.031509 19462 raft_consensus.cc:2454] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae87126d87c14c1381593b355746c1a2 in term 0.
I20260928 05:15:39.032249 19265 leader_election.cc:302] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87126d87c14c1381593b355746c1a2, af3495dc171a4370baa4a67eda68a47c; no voters:
I20260928 05:15:39.032414 19592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e09cf667a4d46e6b224fe5c71a30f1c" candidate_uuid: "ae87126d87c14c1381593b355746c1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd85b5f98784d1d9013c7efece08f14" is_pre_election: true
I20260928 05:15:39.032724 19661 raft_consensus.cc:2781] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:15:39.032896 19661 raft_consensus.cc:482] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:15:39.033023 19661 raft_consensus.cc:3037] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:39.032974 19592 raft_consensus.cc:2454] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae87126d87c14c1381593b355746c1a2 in term 0.
I20260928 05:15:39.035544 19661 raft_consensus.cc:504] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:39.036137 19661 leader_election.cc:288] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [CANDIDATE]: Term 1 election: Requested vote from peers af3495dc171a4370baa4a67eda68a47c (127.0.97.2:44439), edd85b5f98784d1d9013c7efece08f14 (127.0.97.3:44863)
I20260928 05:15:39.036600 19462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e09cf667a4d46e6b224fe5c71a30f1c" candidate_uuid: "ae87126d87c14c1381593b355746c1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3495dc171a4370baa4a67eda68a47c"
I20260928 05:15:39.036657 19592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e09cf667a4d46e6b224fe5c71a30f1c" candidate_uuid: "ae87126d87c14c1381593b355746c1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd85b5f98784d1d9013c7efece08f14"
I20260928 05:15:39.036787 19462 raft_consensus.cc:3037] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:39.036957 19592 raft_consensus.cc:3037] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:39.039134 19462 raft_consensus.cc:2454] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae87126d87c14c1381593b355746c1a2 in term 1.
I20260928 05:15:39.039410 19592 raft_consensus.cc:2454] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae87126d87c14c1381593b355746c1a2 in term 1.
I20260928 05:15:39.039528 19265 leader_election.cc:302] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae87126d87c14c1381593b355746c1a2, af3495dc171a4370baa4a67eda68a47c; no voters:
I20260928 05:15:39.039844 19661 raft_consensus.cc:2781] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:39.040300 19661 raft_consensus.cc:686] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 1 LEADER]: Becoming Leader. State: Replica: ae87126d87c14c1381593b355746c1a2, State: Running, Role: LEADER
I20260928 05:15:39.040807 19661 consensus_queue.cc:237] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } }
I20260928 05:15:39.045670 19188 catalog_manager.cc:5495] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae87126d87c14c1381593b355746c1a2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ae87126d87c14c1381593b355746c1a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae87126d87c14c1381593b355746c1a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 } health_report { overall_health: UNKNOWN } } }
W20260928 05:15:39.054633 19508 tablet.cc:2369] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:39.090121 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.084543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.97.1:36617)
0928 05:15:39.084751 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:39.084755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:39.084773 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:15:39.084875 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:39.084879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:39.084971 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:39.085096 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:15:39.085484 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.086181 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.087780 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.088188 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.088284 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.088768 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:15:39.089255 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:15:39.089381 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:39.089745 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 05:15:39.089795 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:15:39.098336 19462 raft_consensus.cc:1261] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 1 FOLLOWER]: Refusing update from remote peer ae87126d87c14c1381593b355746c1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:39.098423 19592 raft_consensus.cc:1261] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 1 FOLLOWER]: Refusing update from remote peer ae87126d87c14c1381593b355746c1a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:39.099356 19666 consensus_queue.cc:1035] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd85b5f98784d1d9013c7efece08f14" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:39.099699 19661 consensus_queue.cc:1035] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3495dc171a4370baa4a67eda68a47c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 05:15:39.103085 19377 tablet.cc:2369] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:39.110764 19670 mvcc.cc:205] Tried to move back new op lower bound from 7334185120137363456 to 7334185119914975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334185120137363456}]
I20260928 05:15:39.116158 19671 mvcc.cc:205] Tried to move back new op lower bound from 7334185120137363456 to 7334185119914975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334185120137363456}]
I20260928 05:15:39.122893 19189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:39.123155 19189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:39.127014 19189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0fd5491fe0b4ebeb0b571282ed89a7f: Sending DeleteTablet for 3 replicas of tablet 8e09cf667a4d46e6b224fe5c71a30f1c
I20260928 05:15:39.128170 19311 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e09cf667a4d46e6b224fe5c71a30f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:39 UTC) from {username='slave'} at 127.0.0.1:58180
I20260928 05:15:39.128381 19572 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e09cf667a4d46e6b224fe5c71a30f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:39 UTC) from {username='slave'} at 127.0.0.1:60972
I20260928 05:15:39.128468 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e09cf667a4d46e6b224fe5c71a30f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:39 UTC) from {username='slave'} at 127.0.0.1:42674
I20260928 05:15:39.130681 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19249
I20260928 05:15:39.140611 19679 tablet_replica.cc:330] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: stopping tablet replica
I20260928 05:15:39.141178 19679 raft_consensus.cc:2229] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:15:39.141973 19679 raft_consensus.cc:2258] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:39.144335 19679 ts_tablet_manager.cc:1905] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:39.152153 19679 ts_tablet_manager.cc:1918] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:39.152489 19679 log.cc:1196] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/8e09cf667a4d46e6b224fe5c71a30f1c
I20260928 05:15:39.153158 19679 ts_tablet_manager.cc:1939] T 8e09cf667a4d46e6b224fe5c71a30f1c P ae87126d87c14c1381593b355746c1a2: Deleting consensus metadata
I20260928 05:15:39.155411 19173 catalog_manager.cc:4841] TS ae87126d87c14c1381593b355746c1a2 (127.0.97.1:36617): tablet 8e09cf667a4d46e6b224fe5c71a30f1c (table test-table [id=1e731546094c4b5d851dcc5738b26228]) successfully deleted
I20260928 05:15:39.159905 19680 tablet_replica.cc:330] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: stopping tablet replica
I20260928 05:15:39.160521 19680 raft_consensus.cc:2229] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:39.161204 19680 raft_consensus.cc:2258] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:39.165122 19681 tablet_replica.cc:330] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: stopping tablet replica
I20260928 05:15:39.165441 19680 ts_tablet_manager.cc:1905] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:39.165542 19681 raft_consensus.cc:2229] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:39.166026 19681 raft_consensus.cc:2258] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:39.166859 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.132380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.0.97.1:36617)
0928 05:15:39.132625 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:39.132635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:39.157993 (+ 25358us) server_negotiation.cc:407] Connection header received
0928 05:15:39.158234 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:39.158240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:39.158347 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:39.158455 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:39.159140 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.160353 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.162584 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.163267 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.163370 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.164946 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:39.164963 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:39.164980 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:39.165771 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:39.165835 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:39.165859 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:39.165946 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:39.166331 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 05:15:39.166437 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:15:39.168099 19681 ts_tablet_manager.cc:1905] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:39.168197 19371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:39.173091 19680 ts_tablet_manager.cc:1918] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:39.173265 19680 log.cc:1196] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/8e09cf667a4d46e6b224fe5c71a30f1c
I20260928 05:15:39.173799 19680 ts_tablet_manager.cc:1939] T 8e09cf667a4d46e6b224fe5c71a30f1c P edd85b5f98784d1d9013c7efece08f14: Deleting consensus metadata
I20260928 05:15:39.175029 19681 ts_tablet_manager.cc:1918] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:39.175189 19681 log.cc:1196] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/8e09cf667a4d46e6b224fe5c71a30f1c
I20260928 05:15:39.175717 19681 ts_tablet_manager.cc:1939] T 8e09cf667a4d46e6b224fe5c71a30f1c P af3495dc171a4370baa4a67eda68a47c: Deleting consensus metadata
I20260928 05:15:39.175828 19174 catalog_manager.cc:4841] TS edd85b5f98784d1d9013c7efece08f14 (127.0.97.3:44863): tablet 8e09cf667a4d46e6b224fe5c71a30f1c (table test-table [id=1e731546094c4b5d851dcc5738b26228]) successfully deleted
I20260928 05:15:39.177186 19174 catalog_manager.cc:4841] TS af3495dc171a4370baa4a67eda68a47c (127.0.97.2:44439): tablet 8e09cf667a4d46e6b224fe5c71a30f1c (table test-table [id=1e731546094c4b5d851dcc5738b26228]) successfully deleted
I20260928 05:15:39.268186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19249
I20260928 05:15:39.287124 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19380
I20260928 05:15:39.294137 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.287950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.0.97.2:44439)
0928 05:15:39.288145 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:39.288151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:39.288172 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:15:39.288280 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:39.288284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:39.288416 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:39.288531 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:39.288934 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.289710 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.291144 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.291567 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.291704 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.292762 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:39.292771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:39.292786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:39.293362 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:39.293420 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:39.293440 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:39.293576 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:39.293703 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 05:15:39.293806 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:15:39.294958 19502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:39.386865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19380
I20260928 05:15:39.403873 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19510
I20260928 05:15:39.410627 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:39.404662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.0.97.3:44863)
0928 05:15:39.404822 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:39.404826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:39.404844 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:15:39.404960 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:39.404964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:39.405048 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:39.405153 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:39.405539 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.406225 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.407629 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:39.408078 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:39.408219 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:39.409277 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:39.409286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:39.409301 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:39.409859 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:39.409906 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:39.409953 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:39.410053 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:39.410208 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 05:15:39.410295 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:15:39.411247 19632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:39.499087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19510
I20260928 05:15:39.515240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19158
I20260928 05:15:39.516309 19219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:39.624532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19158
2026-09-28T05:15:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2822 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:39 dist-test-slave-6718 krb5kdc[19689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:39 dist-test-slave-6718 krb5kdc[19689](info): set up 2 sockets
Sep 28 05:15:39 dist-test-slave-6718 krb5kdc[19689](info): commencing operation
krb5kdc: starting...
W20260928 05:15:40.734920 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:40 dist-test-slave-6718 krb5kdc[19689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:15:40.983740 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42559
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:41.311512 19702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:41.311862 19702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:41.312043 19702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:41.320883 19702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:41.321023 19702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:41.321126 19702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:41.321233 19702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:41.321317 19702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:41.331280 19702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:41.332612 19702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:41.333958 19702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:41.346657 19707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:41.346686 19708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:41.346616 19710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:41.347492 19702 server_base.cc:1034] running on GCE node
I20260928 05:15:41.348260 19702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:41.349596 19702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:41.350792 19702 hybrid_clock.cc:648] HybridClock initialized: now 1790572541350742 us; error 38 us; skew 500 ppm
I20260928 05:15:41.351163 19702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:41 dist-test-slave-6718 krb5kdc[19689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:41.361060 19702 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:15:41.362820 19702 webserver.cc:460] Webserver started at http://127.0.97.62:41543/ using document root <none> and password file <none>
I20260928 05:15:41.363584 19702 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:41.363722 19702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:41.364061 19702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:41.367259 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "3d0213f6003e4c879115fb91ca6ccbc5"
format_stamp: "Formatted at 2026-09-28 05:15:41 on dist-test-slave-6718"
server_key: "89373e40074d2d3ff185f5982cf5750e"
server_key_iv: "a2669761a0a66e2a102cfd4a1696fb20"
server_key_version: "encryptionkey@0"
I20260928 05:15:41.368011 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d0213f6003e4c879115fb91ca6ccbc5"
format_stamp: "Formatted at 2026-09-28 05:15:41 on dist-test-slave-6718"
server_key: "89373e40074d2d3ff185f5982cf5750e"
server_key_iv: "a2669761a0a66e2a102cfd4a1696fb20"
server_key_version: "encryptionkey@0"
I20260928 05:15:41.373164 19702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 05:15:41.376739 19717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:41.378334 19702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:15:41.378528 19702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "3d0213f6003e4c879115fb91ca6ccbc5"
format_stamp: "Formatted at 2026-09-28 05:15:41 on dist-test-slave-6718"
server_key: "89373e40074d2d3ff185f5982cf5750e"
server_key_iv: "a2669761a0a66e2a102cfd4a1696fb20"
server_key_version: "encryptionkey@0"
I20260928 05:15:41.378782 19702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:41.396539 19702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:41.401001 19702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:41.401307 19702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:41.419720 19702 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42559
I20260928 05:15:41.419768 19768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42559 every 8 connection(s)
I20260928 05:15:41.421182 19702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:41.424971 19769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:41.427731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19702
I20260928 05:15:41.427920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:41.428341 388 external_mini_cluster.cc:1428] Setting key a31d146a2d670715dbafdfb206df5f24
I20260928 05:15:41.438665 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5: Bootstrap starting.
I20260928 05:15:41.442198 19769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:41.443269 19769 log.cc:824] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5: Log is configured to *not* fsync() on all Append() calls
Sep 28 05:15:41 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:41.446724 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5: No bootstrap required, opened a new log
I20260928 05:15:41.454331 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:41.430401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.0.97.62:42559)
0928 05:15:41.431323 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:41.431339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:41.431395 (+ 56us) server_negotiation.cc:407] Connection header received
0928 05:15:41.433731 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:41.433759 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:41.434267 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:41.434781 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:41.436153 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:41.438098 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:41.439486 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:41.440566 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:41.440752 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:41.445125 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:41.445137 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:41.445155 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:41.445196 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:41.449185 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:41.449995 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:41.450004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:41.450024 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:41.450226 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:41.450904 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:41.450910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:41.450914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:41.451706 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:41.452025 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:41.452465 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 05:15:41.452941 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":352,"threads_started":1}
I20260928 05:15:41.457435 19769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260928 05:15:41.457958 19769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:41.458093 19769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0213f6003e4c879115fb91ca6ccbc5, State: Initialized, Role: FOLLOWER
I20260928 05:15:41.458928 19769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260928 05:15:41.459280 19769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:41.459460 19769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:41.459625 19769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:41.463243 19769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260928 05:15:41.463791 19769 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0213f6003e4c879115fb91ca6ccbc5; no voters:
I20260928 05:15:41.464442 19769 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:41.464666 19774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:41.466187 19774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0213f6003e4c879115fb91ca6ccbc5, State: Running, Role: LEADER
I20260928 05:15:41.466812 19769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:41.466801 19774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } }
I20260928 05:15:41.469455 19776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0213f6003e4c879115fb91ca6ccbc5. Latest consensus state: current_term: 1 leader_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } } }
I20260928 05:15:41.469735 19776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:41.470243 19775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0213f6003e4c879115fb91ca6ccbc5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42559 } } }
I20260928 05:15:41.470501 19775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:41.470793 19780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:41.477674 19780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:41.487105 19780 catalog_manager.cc:1347] Generated new cluster ID: 931dab3971664b768cf487c94b1b0ffd
I20260928 05:15:41.487216 19780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:41.515779 19780 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:41.517110 19780 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:41.528441 19780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5: Generated new TSK 0
I20260928 05:15:41.529326 19780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 05:15:41.590287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42559
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:41.937642 19795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:41.937968 19795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:41.938130 19795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:41.946110 19795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:41.946235 19795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:41.946456 19795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:41.957923 19795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:41.959357 19795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:41.960690 19795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:41.974005 19803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:41.973961 19800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:41.974027 19801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:41.975837 19795 server_base.cc:1034] running on GCE node
I20260928 05:15:41.976550 19795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:41.977788 19795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:41.978999 19795 hybrid_clock.cc:648] HybridClock initialized: now 1790572541978951 us; error 42 us; skew 500 ppm
I20260928 05:15:41.979337 19795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:41 dist-test-slave-6718 krb5kdc[19689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:41.982734 19795 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 05:15:41.984380 19795 webserver.cc:460] Webserver started at http://127.0.97.1:34823/ using document root <none> and password file <none>
I20260928 05:15:41.985085 19795 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:41.985185 19795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:41.985460 19795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:41.988693 19795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "de3ef9a4dce240ac9e9dc538c1a7b797"
format_stamp: "Formatted at 2026-09-28 05:15:41 on dist-test-slave-6718"
server_key: "ee23b0bdc5e3ee25315a3a86de110583"
server_key_iv: "12df08c2301d55b5ffa2c3085fae1d38"
server_key_version: "encryptionkey@0"
I20260928 05:15:41.989451 19795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "de3ef9a4dce240ac9e9dc538c1a7b797"
format_stamp: "Formatted at 2026-09-28 05:15:41 on dist-test-slave-6718"
server_key: "ee23b0bdc5e3ee25315a3a86de110583"
server_key_iv: "12df08c2301d55b5ffa2c3085fae1d38"
server_key_version: "encryptionkey@0"
I20260928 05:15:41.994700 19795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:15:41.998354 19810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:41.999797 19795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:42.000008 19795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "de3ef9a4dce240ac9e9dc538c1a7b797"
format_stamp: "Formatted at 2026-09-28 05:15:41 on dist-test-slave-6718"
server_key: "ee23b0bdc5e3ee25315a3a86de110583"
server_key_iv: "12df08c2301d55b5ffa2c3085fae1d38"
server_key_version: "encryptionkey@0"
I20260928 05:15:42.000232 19795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:42.022993 19795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:42.027740 19795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:42.028095 19795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:42.029140 19795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:42.030634 19795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:42.030735 19795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:42.030864 19795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:42.030936 19795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:42.064078 19795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41553
I20260928 05:15:42.064170 19922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41553 every 8 connection(s)
I20260928 05:15:42.065420 19795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:42.067348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19795
I20260928 05:15:42.067560 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:15:42.068069 388 external_mini_cluster.cc:1428] Setting key c4099a97efc9c40f1b7010acf43b2fa9
Sep 28 05:15:42 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:42.087101 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:42.068307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34551 (local address 127.0.97.62:42559)
0928 05:15:42.068788 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:42.068794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:42.070022 (+ 1228us) server_negotiation.cc:407] Connection header received
0928 05:15:42.072062 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:42.072067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:42.072174 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:42.072304 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:42.073789 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.074521 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.076087 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.076505 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.076642 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:42.080627 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:42.080633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:42.080645 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:42.080678 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:42.083678 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:42.084518 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:42.084529 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:42.084531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:42.084658 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:42.085152 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:42.085163 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:42.085166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:42.085453 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:42.085621 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:42.086395 (+ 774us) server_negotiation.cc:299] Negotiation successful
0928 05:15:42.086676 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260928 05:15:42.088135 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:42.069002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42559 (local address 127.0.97.1:34551)
0928 05:15:42.069863 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:42.069910 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:42.071793 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:42.072551 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:42.072560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:42.073057 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:42.073619 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:42.073636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.074707 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.074712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:42.075927 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:42.075938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.076747 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.076757 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:42.076997 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:42.077720 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:42.077742 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:42.080394 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:42.083849 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:42.083859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:42.083877 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:42.084356 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:42.084783 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:42.084788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:42.084793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:42.085012 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:42.085594 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:42.085603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:42.085991 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 05:15:42.086290 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 05:15:42.086940 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":355,"threads_started":1}
I20260928 05:15:42.089896 19923 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42559
I20260928 05:15:42.090257 19923 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:42.091226 19923 heartbeater.cc:511] Master 127.0.97.62:42559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 05:15:42.093693 19734 ts_manager.cc:194] Registered new tserver with Master: de3ef9a4dce240ac9e9dc538c1a7b797 (127.0.97.1:41553)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 05:15:42.095299 19734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 05:15:42.126636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42559
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:42.453117 19929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:42.453395 19929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:42.453557 19929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:42.460722 19929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:42.460829 19929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:42.461019 19929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:42.470718 19929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:42.471989 19929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:42.473268 19929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:42.487737 19935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:42.487788 19937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:42.487813 19934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:42.488670 19929 server_base.cc:1034] running on GCE node
I20260928 05:15:42.489331 19929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:42.490249 19929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:42.491478 19929 hybrid_clock.cc:648] HybridClock initialized: now 1790572542491439 us; error 54 us; skew 500 ppm
I20260928 05:15:42.491787 19929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:42 dist-test-slave-6718 krb5kdc[19689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:42.494969 19929 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 05:15:42.496299 19929 webserver.cc:460] Webserver started at http://127.0.97.2:36843/ using document root <none> and password file <none>
I20260928 05:15:42.496956 19929 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:42.497053 19929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:42.497298 19929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:42.500408 19929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "ec400e30ce944a57b774e7e8e3e8c991"
format_stamp: "Formatted at 2026-09-28 05:15:42 on dist-test-slave-6718"
server_key: "911418f41ef7e6aa055971ffed5e572b"
server_key_iv: "61311cf49309d828b6670659b2f76a41"
server_key_version: "encryptionkey@0"
I20260928 05:15:42.501132 19929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ec400e30ce944a57b774e7e8e3e8c991"
format_stamp: "Formatted at 2026-09-28 05:15:42 on dist-test-slave-6718"
server_key: "911418f41ef7e6aa055971ffed5e572b"
server_key_iv: "61311cf49309d828b6670659b2f76a41"
server_key_version: "encryptionkey@0"
I20260928 05:15:42.506301 19929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 05:15:42.510116 19944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:42.511574 19929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:42.511709 19929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "ec400e30ce944a57b774e7e8e3e8c991"
format_stamp: "Formatted at 2026-09-28 05:15:42 on dist-test-slave-6718"
server_key: "911418f41ef7e6aa055971ffed5e572b"
server_key_iv: "61311cf49309d828b6670659b2f76a41"
server_key_version: "encryptionkey@0"
I20260928 05:15:42.511893 19929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:42.545100 19929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:42.549392 19929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:42.549638 19929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:42.550716 19929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:42.552018 19929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:42.552089 19929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:42.552176 19929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:42.552220 19929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:42.585153 19929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42327
I20260928 05:15:42.585160 20056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42327 every 8 connection(s)
I20260928 05:15:42.586433 19929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:42.590975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 19929
I20260928 05:15:42.591156 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:15:42.591589 388 external_mini_cluster.cc:1428] Setting key bb3e32de34ddcc802f735bd5c7747d01
Sep 28 05:15:42 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:42.606323 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:42.588945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57617 (local address 127.0.97.62:42559)
0928 05:15:42.589391 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:42.589395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:42.590337 (+ 942us) server_negotiation.cc:407] Connection header received
0928 05:15:42.592321 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:42.592327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:42.592415 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:42.592554 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:42.593978 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.594771 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.596281 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.596700 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.596834 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:42.600391 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:42.600398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:42.600403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:42.600442 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:42.603117 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:42.603851 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:42.603858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:42.603862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:42.604066 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:42.604497 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:42.604503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:42.604506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:42.604781 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:42.604950 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:42.605710 (+ 760us) server_negotiation.cc:299] Negotiation successful
0928 05:15:42.605941 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260928 05:15:42.607362 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:42.589516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42559 (local address 127.0.97.2:57617)
0928 05:15:42.590177 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:42.590223 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:42.592056 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:42.592789 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:42.592797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:42.593286 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:42.593815 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:42.593830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.594917 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.594921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:42.596130 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:42.596142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:42.596922 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:42.596938 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:42.597165 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:42.597872 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:42.597903 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:42.600150 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:42.603263 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:42.603272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:42.603289 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:42.603722 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:42.604178 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:42.604184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:42.604188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:42.604376 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:42.604912 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:42.604920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:42.605343 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 05:15:42.605624 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 05:15:42.606182 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260928 05:15:42.608868 20057 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42559
I20260928 05:15:42.609200 20057 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:42.609925 20057 heartbeater.cc:511] Master 127.0.97.62:42559 requested a full tablet report, sending...
I20260928 05:15:42.611531 19734 ts_manager.cc:194] Registered new tserver with Master: ec400e30ce944a57b774e7e8e3e8c991 (127.0.97.2:42327)
I20260928 05:15:42.612861 19734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57617
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 05:15:42.646656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42559
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:42.976455 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:42.976732 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:42.976825 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:42.984375 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:42.984488 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:42.984601 20063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:42.994189 20063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:42.995582 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:42.996856 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:43.009851 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:43.010113 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:43.010221 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:43.010902 20063 server_base.cc:1034] running on GCE node
I20260928 05:15:43.011509 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:43.012387 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:43.013597 20063 hybrid_clock.cc:648] HybridClock initialized: now 1790572543013552 us; error 34 us; skew 500 ppm
I20260928 05:15:43.013906 20063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:43 dist-test-slave-6718 krb5kdc[19689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:43.017160 20063 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 05:15:43.018635 20063 webserver.cc:460] Webserver started at http://127.0.97.3:41177/ using document root <none> and password file <none>
I20260928 05:15:43.019309 20063 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:43.019433 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:43.019795 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:43.022735 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "e4f88f4242f6486bb8ae26866212e5a4"
format_stamp: "Formatted at 2026-09-28 05:15:43 on dist-test-slave-6718"
server_key: "784801320dc582dba824ed7b8d1a4ea8"
server_key_iv: "e84a7ba4a3f175928c33ca1d1c8779ef"
server_key_version: "encryptionkey@0"
I20260928 05:15:43.023460 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e4f88f4242f6486bb8ae26866212e5a4"
format_stamp: "Formatted at 2026-09-28 05:15:43 on dist-test-slave-6718"
server_key: "784801320dc582dba824ed7b8d1a4ea8"
server_key_iv: "e84a7ba4a3f175928c33ca1d1c8779ef"
server_key_version: "encryptionkey@0"
I20260928 05:15:43.028441 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:15:43.032014 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:43.033561 20063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:15:43.033733 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "e4f88f4242f6486bb8ae26866212e5a4"
format_stamp: "Formatted at 2026-09-28 05:15:43 on dist-test-slave-6718"
server_key: "784801320dc582dba824ed7b8d1a4ea8"
server_key_iv: "e84a7ba4a3f175928c33ca1d1c8779ef"
server_key_version: "encryptionkey@0"
I20260928 05:15:43.033932 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:43.050408 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:43.055382 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:43.055719 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:43.056770 20063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:43.058225 20063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:43.058350 20063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:43.058517 20063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:43.058678 20063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 05:15:43.093892 20063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44719
I20260928 05:15:43.093930 20190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44719 every 8 connection(s)
I20260928 05:15:43.095398 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:43.098248 19923 heartbeater.cc:503] Master 127.0.97.62:42559 was elected leader, sending a full tablet report...
I20260928 05:15:43.101276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20063
I20260928 05:15:43.101451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:15:43.101876 388 external_mini_cluster.cc:1428] Setting key 52622b1827efa8f1820ec751a7306482
Sep 28 05:15:43 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:15:43.116084 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.098178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44609 (local address 127.0.97.62:42559)
0928 05:15:43.098376 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.098383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.099739 (+ 1356us) server_negotiation.cc:407] Connection header received
0928 05:15:43.101712 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.101717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.101804 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.101933 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:43.103708 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.104693 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.106298 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.106778 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.106913 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.110591 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:43.110597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:43.110600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:43.110633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:43.113084 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.113740 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.113743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.113746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.113859 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.114240 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.114243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.114245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.114496 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:43.114713 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.115525 (+ 812us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.115761 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:15:43.117172 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.098765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42559 (local address 127.0.97.3:44609)
0928 05:15:43.099601 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:43.099645 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:43.101410 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:43.102166 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:43.102174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:43.102831 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:43.103543 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.103562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.104840 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.104845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.106154 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.106166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.107015 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.107022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.107261 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.107949 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:15:43.107970 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:43.110348 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:43.113205 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:43.113212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:43.113225 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:43.113641 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:43.113964 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:15:43.113967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:15:43.113969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:15:43.114152 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:15:43.114692 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:43.114698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:43.115049 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 05:15:43.115410 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 05:15:43.115952 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":364,"threads_started":1}
I20260928 05:15:43.118649 20191 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42559
I20260928 05:15:43.118943 20191 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:43.119690 20191 heartbeater.cc:511] Master 127.0.97.62:42559 requested a full tablet report, sending...
I20260928 05:15:43.121132 19734 ts_manager.cc:194] Registered new tserver with Master: e4f88f4242f6486bb8ae26866212e5a4 (127.0.97.3:44719)
I20260928 05:15:43.121949 19734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44609
I20260928 05:15:43.129418 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:15:43.146255 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.133628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.97.62:42559)
0928 05:15:43.133808 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.133812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.134311 (+ 499us) server_negotiation.cc:407] Connection header received
0928 05:15:43.134428 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.134433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.134504 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.134649 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:43.135921 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.136650 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.138015 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.138663 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.138783 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.141127 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:43.141131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:43.141135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:43.141165 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:43.143813 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.144469 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.144473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.144475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.144598 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.145024 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.145028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.145030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.145327 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:43.145550 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.145837 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.146015 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:15:43.150175 19734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:15:43.157905 19734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:15:43.181787 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.171440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.0.97.1:41553)
0928 05:15:43.172358 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.172368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.172471 (+ 103us) server_negotiation.cc:407] Connection header received
0928 05:15:43.172855 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.172868 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.173195 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.173438 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:43.174572 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.176323 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.179144 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.180604 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.180744 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.180993 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.181286 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.181445 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":572,"threads_started":1}
I20260928 05:15:43.181835 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.171053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:38186)
0928 05:15:43.172307 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:43.172348 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:43.172571 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:43.173767 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:43.173779 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:43.173824 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:43.174320 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.174335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.176500 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.176505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.178951 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.178966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.180795 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.180802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.181055 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.181093 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 05:15:43.181235 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 05:15:43.181442 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":1,"thread_start_us":770,"threads_started":1}
I20260928 05:15:43.183444 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.172681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44719 (local address 127.0.0.1:60388)
0928 05:15:43.174132 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:43.174157 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:43.174308 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:43.175389 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:43.175395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:43.175421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:43.175951 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.175966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.177976 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.177981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.180285 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.180301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.182616 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.182622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.182803 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.182836 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:15:43.182948 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 05:15:43.183087 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":1140,"threads_started":1}
I20260928 05:15:43.184374 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.173027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.97.3:44719)
0928 05:15:43.173832 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.173844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.174452 (+ 608us) server_negotiation.cc:407] Connection header received
0928 05:15:43.174594 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.174604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.174848 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.175066 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:43.176150 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.177790 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.180481 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.182049 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.182188 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.182448 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.183830 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.184018 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":451,"threads_started":1}
I20260928 05:15:43.184785 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.174397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.0.97.2:42327)
0928 05:15:43.175586 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.175596 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.176259 (+ 663us) server_negotiation.cc:407] Connection header received
0928 05:15:43.176439 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.176447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.176621 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.176809 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:43.177503 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.179175 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.181767 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.183792 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.183941 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.184122 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.184352 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.184501 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":869,"threads_started":1}
I20260928 05:15:43.184813 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.174828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42327 (local address 127.0.0.1:54436)
0928 05:15:43.176134 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:43.176151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:43.176295 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:43.176800 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:43.176805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:43.176837 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:43.177332 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.177341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.179315 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.179324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.181556 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.181576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.183978 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.183984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.184163 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.184198 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 05:15:43.184301 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 05:15:43.184423 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":96,"thread_start_us":958,"threads_started":1}
I20260928 05:15:43.186597 19858 tablet_service.cc:1467] Processing CreateTablet for tablet 3a194312a2444f879dbb6cf37d9ca4cb (DEFAULT_TABLE table=test-table [id=688f9ce9f38e4e3989a65ac824d279c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:43.188320 19858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a194312a2444f879dbb6cf37d9ca4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:43.188210 20126 tablet_service.cc:1467] Processing CreateTablet for tablet 3a194312a2444f879dbb6cf37d9ca4cb (DEFAULT_TABLE table=test-table [id=688f9ce9f38e4e3989a65ac824d279c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:43.189000 19992 tablet_service.cc:1467] Processing CreateTablet for tablet 3a194312a2444f879dbb6cf37d9ca4cb (DEFAULT_TABLE table=test-table [id=688f9ce9f38e4e3989a65ac824d279c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:43.189670 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a194312a2444f879dbb6cf37d9ca4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:43.190312 19992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a194312a2444f879dbb6cf37d9ca4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:43.203488 20209 tablet_bootstrap.cc:492] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Bootstrap starting.
I20260928 05:15:43.204663 20210 tablet_bootstrap.cc:492] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Bootstrap starting.
I20260928 05:15:43.207266 20211 tablet_bootstrap.cc:492] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Bootstrap starting.
I20260928 05:15:43.208421 20209 tablet_bootstrap.cc:654] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:43.208432 20210 tablet_bootstrap.cc:654] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:43.209882 20210 log.cc:824] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:43.209888 20209 log.cc:824] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:43.210266 20211 tablet_bootstrap.cc:654] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:43.211651 20211 log.cc:824] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:43.213104 20210 tablet_bootstrap.cc:492] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: No bootstrap required, opened a new log
I20260928 05:15:43.213482 20210 ts_tablet_manager.cc:1397] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 05:15:43.215245 20211 tablet_bootstrap.cc:492] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: No bootstrap required, opened a new log
I20260928 05:15:43.215516 20211 ts_tablet_manager.cc:1397] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:15:43.219460 20209 tablet_bootstrap.cc:492] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: No bootstrap required, opened a new log
I20260928 05:15:43.219767 20209 ts_tablet_manager.cc:1397] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.011s
I20260928 05:15:43.224411 20210 raft_consensus.cc:348] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.224946 20210 raft_consensus.cc:374] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:43.225082 20210 raft_consensus.cc:729] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f88f4242f6486bb8ae26866212e5a4, State: Initialized, Role: FOLLOWER
I20260928 05:15:43.225773 20210 consensus_queue.cc:260] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.226233 20211 raft_consensus.cc:348] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.227020 20211 raft_consensus.cc:374] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:43.227084 20191 heartbeater.cc:503] Master 127.0.97.62:42559 was elected leader, sending a full tablet report...
I20260928 05:15:43.227166 20211 raft_consensus.cc:729] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec400e30ce944a57b774e7e8e3e8c991, State: Initialized, Role: FOLLOWER
I20260928 05:15:43.227741 20210 ts_tablet_manager.cc:1428] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260928 05:15:43.227893 20211 consensus_queue.cc:260] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.229090 20057 heartbeater.cc:503] Master 127.0.97.62:42559 was elected leader, sending a full tablet report...
I20260928 05:15:43.229780 20211 ts_tablet_manager.cc:1428] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 05:15:43.229923 20209 raft_consensus.cc:348] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.230485 20209 raft_consensus.cc:374] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:43.230633 20209 raft_consensus.cc:729] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3ef9a4dce240ac9e9dc538c1a7b797, State: Initialized, Role: FOLLOWER
I20260928 05:15:43.231458 20209 consensus_queue.cc:260] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.239847 20209 ts_tablet_manager.cc:1428] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
W20260928 05:15:43.317850 19924 tablet.cc:2369] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:15:43.338522 20058 tablet.cc:2369] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:15:43.347131 20192 tablet.cc:2369] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:15:43.434784 20215 raft_consensus.cc:482] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:43.435105 20215 raft_consensus.cc:504] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.436895 20215 leader_election.cc:288] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de3ef9a4dce240ac9e9dc538c1a7b797 (127.0.97.1:41553), ec400e30ce944a57b774e7e8e3e8c991 (127.0.97.2:42327)
I20260928 05:15:43.442728 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.437374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45423 (local address 127.0.97.1:41553)
0928 05:15:43.437617 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.437625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.437651 (+ 26us) server_negotiation.cc:407] Connection header received
0928 05:15:43.437865 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.437871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.438018 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.438125 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:43.438583 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.439320 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.441056 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.441819 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.441978 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.442076 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.442331 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.442455 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 05:15:43.442737 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.437365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.97.3:45423)
0928 05:15:43.437589 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:43.437606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:43.437712 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:43.438144 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:43.438149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:43.438179 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:43.438399 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.438415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.439486 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.439491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.440892 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.440907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.441986 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.441991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.442178 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.442201 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:15:43.442306 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:15:43.442444 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 05:15:43.443831 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.437838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42327 (local address 127.0.97.3:47917)
0928 05:15:43.438800 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:43.438814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:43.438948 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:43.439266 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:43.439270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:43.439292 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:43.439667 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.439676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.440661 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.440667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.441950 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:43.441961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.443244 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.443250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:43.443417 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.443448 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 05:15:43.443523 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 05:15:43.443607 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":677,"threads_started":1}
I20260928 05:15:43.443972 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.438099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47917 (local address 127.0.97.2:42327)
0928 05:15:43.438294 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.438300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.438917 (+ 617us) server_negotiation.cc:407] Connection header received
0928 05:15:43.439097 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.439101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.439157 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.439252 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:43.439826 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.440517 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.442105 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.443042 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.443167 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.443237 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.443580 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.443708 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:15:43.443847 19878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a194312a2444f879dbb6cf37d9ca4cb" candidate_uuid: "e4f88f4242f6486bb8ae26866212e5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" is_pre_election: true
I20260928 05:15:43.444476 19878 raft_consensus.cc:2454] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f88f4242f6486bb8ae26866212e5a4 in term 0.
I20260928 05:15:43.444953 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a194312a2444f879dbb6cf37d9ca4cb" candidate_uuid: "e4f88f4242f6486bb8ae26866212e5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec400e30ce944a57b774e7e8e3e8c991" is_pre_election: true
I20260928 05:15:43.445165 20081 leader_election.cc:302] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de3ef9a4dce240ac9e9dc538c1a7b797, e4f88f4242f6486bb8ae26866212e5a4; no voters:
I20260928 05:15:43.445513 20012 raft_consensus.cc:2454] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f88f4242f6486bb8ae26866212e5a4 in term 0.
I20260928 05:15:43.445571 20215 raft_consensus.cc:2781] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:15:43.445704 20215 raft_consensus.cc:482] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:15:43.445807 20215 raft_consensus.cc:3037] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:43.448099 20215 raft_consensus.cc:504] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.448685 20215 leader_election.cc:288] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [CANDIDATE]: Term 1 election: Requested vote from peers de3ef9a4dce240ac9e9dc538c1a7b797 (127.0.97.1:41553), ec400e30ce944a57b774e7e8e3e8c991 (127.0.97.2:42327)
I20260928 05:15:43.449242 19878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a194312a2444f879dbb6cf37d9ca4cb" candidate_uuid: "e4f88f4242f6486bb8ae26866212e5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797"
I20260928 05:15:43.449282 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a194312a2444f879dbb6cf37d9ca4cb" candidate_uuid: "e4f88f4242f6486bb8ae26866212e5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec400e30ce944a57b774e7e8e3e8c991"
I20260928 05:15:43.449477 19878 raft_consensus.cc:3037] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:43.449549 20012 raft_consensus.cc:3037] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:43.451808 19878 raft_consensus.cc:2454] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f88f4242f6486bb8ae26866212e5a4 in term 1.
I20260928 05:15:43.451817 20012 raft_consensus.cc:2454] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f88f4242f6486bb8ae26866212e5a4 in term 1.
I20260928 05:15:43.452265 20081 leader_election.cc:302] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de3ef9a4dce240ac9e9dc538c1a7b797, e4f88f4242f6486bb8ae26866212e5a4; no voters:
I20260928 05:15:43.452649 20215 raft_consensus.cc:2781] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:43.453135 20215 raft_consensus.cc:686] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 1 LEADER]: Becoming Leader. State: Replica: e4f88f4242f6486bb8ae26866212e5a4, State: Running, Role: LEADER
I20260928 05:15:43.453631 20215 consensus_queue.cc:237] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } }
I20260928 05:15:43.458491 19733 catalog_manager.cc:5495] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4f88f4242f6486bb8ae26866212e5a4 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e4f88f4242f6486bb8ae26866212e5a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f88f4242f6486bb8ae26866212e5a4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 } health_report { overall_health: UNKNOWN } } }
I20260928 05:15:43.479050 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.473404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.97.3:44719)
0928 05:15:43.473639 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.473647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.473741 (+ 94us) server_negotiation.cc:407] Connection header received
0928 05:15:43.473994 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.473998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.474075 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.474230 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:15:43.474653 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.475368 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.476765 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.477170 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.477301 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.477836 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:15:43.478352 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:15:43.478627 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.478680 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.478762 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 05:15:43.487562 19878 raft_consensus.cc:1261] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 1 FOLLOWER]: Refusing update from remote peer e4f88f4242f6486bb8ae26866212e5a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:43.487562 20012 raft_consensus.cc:1261] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 1 FOLLOWER]: Refusing update from remote peer e4f88f4242f6486bb8ae26866212e5a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:43.488577 20219 consensus_queue.cc:1035] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec400e30ce944a57b774e7e8e3e8c991" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:43.488955 20215 consensus_queue.cc:1035] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3ef9a4dce240ac9e9dc538c1a7b797" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:43.504230 20223 mvcc.cc:205] Tried to move back new op lower bound from 7334185138114711552 to 7334185137990021120. Current Snapshot: MvccSnapshot[applied={T|T < 7334185138114711552}]
I20260928 05:15:43.514304 19733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:43.514637 19733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:43.518659 19733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d0213f6003e4c879115fb91ca6ccbc5: Sending DeleteTablet for 3 replicas of tablet 3a194312a2444f879dbb6cf37d9ca4cb
I20260928 05:15:43.519968 19858 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a194312a2444f879dbb6cf37d9ca4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38186
I20260928 05:15:43.520249 20126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a194312a2444f879dbb6cf37d9ca4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60388
I20260928 05:15:43.520975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19795
I20260928 05:15:43.521325 19992 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a194312a2444f879dbb6cf37d9ca4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54436
I20260928 05:15:43.532672 20231 tablet_replica.cc:330] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: stopping tablet replica
I20260928 05:15:43.533443 20231 raft_consensus.cc:2229] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:43.534176 20231 raft_consensus.cc:2258] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:43.536738 20231 ts_tablet_manager.cc:1905] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:43.538952 20232 tablet_replica.cc:330] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: stopping tablet replica
I20260928 05:15:43.541554 20232 raft_consensus.cc:2229] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:15:43.542323 20233 tablet_replica.cc:330] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: stopping tablet replica
I20260928 05:15:43.542354 20232 raft_consensus.cc:2258] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:43.543249 20233 raft_consensus.cc:2229] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:43.543881 20233 raft_consensus.cc:2258] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:43.546046 20232 ts_tablet_manager.cc:1905] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:43.546259 20231 ts_tablet_manager.cc:1918] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 05:15:43 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 05:15:43.546490 20231 log.cc:1196] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-0/wal/wals/3a194312a2444f879dbb6cf37d9ca4cb
I20260928 05:15:43.547132 20231 ts_tablet_manager.cc:1939] T 3a194312a2444f879dbb6cf37d9ca4cb P de3ef9a4dce240ac9e9dc538c1a7b797: Deleting consensus metadata
I20260928 05:15:43.548444 20233 ts_tablet_manager.cc:1905] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:43.549659 19720 catalog_manager.cc:4841] TS de3ef9a4dce240ac9e9dc538c1a7b797 (127.0.97.1:41553): tablet 3a194312a2444f879dbb6cf37d9ca4cb (table test-table [id=688f9ce9f38e4e3989a65ac824d279c2]) successfully deleted
I20260928 05:15:43.555490 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.522337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.0.97.1:41553)
0928 05:15:43.522558 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.522565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.532634 (+ 10069us) server_negotiation.cc:407] Connection header received
0928 05:15:43.533175 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.533182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.533278 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.533363 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:43.535550 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.536697 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.539493 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.540170 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.540294 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.547868 (+ 7574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:43.547886 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:43.547906 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:43.547968 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:43.551900 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.552879 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.552891 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.552896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.553078 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.553539 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.553544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.553548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.554005 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:43.554319 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.554690 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.554925 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 05:15:43.555666 20232 ts_tablet_manager.cc:1918] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:43.556264 19918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:43.562470 20233 ts_tablet_manager.cc:1918] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:43.562737 20233 log.cc:1196] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/3a194312a2444f879dbb6cf37d9ca4cb
I20260928 05:15:43.562737 20232 log.cc:1196] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/3a194312a2444f879dbb6cf37d9ca4cb
I20260928 05:15:43.563351 20232 ts_tablet_manager.cc:1939] T 3a194312a2444f879dbb6cf37d9ca4cb P e4f88f4242f6486bb8ae26866212e5a4: Deleting consensus metadata
I20260928 05:15:43.563349 20233 ts_tablet_manager.cc:1939] T 3a194312a2444f879dbb6cf37d9ca4cb P ec400e30ce944a57b774e7e8e3e8c991: Deleting consensus metadata
I20260928 05:15:43.565264 19720 catalog_manager.cc:4841] TS e4f88f4242f6486bb8ae26866212e5a4 (127.0.97.3:44719): tablet 3a194312a2444f879dbb6cf37d9ca4cb (table test-table [id=688f9ce9f38e4e3989a65ac824d279c2]) successfully deleted
I20260928 05:15:43.565693 19720 catalog_manager.cc:4841] TS ec400e30ce944a57b774e7e8e3e8c991 (127.0.97.2:42327): tablet 3a194312a2444f879dbb6cf37d9ca4cb (table test-table [id=688f9ce9f38e4e3989a65ac824d279c2]) successfully deleted
I20260928 05:15:43.678053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19795
I20260928 05:15:43.696280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19929
Sep 28 05:15:43 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 05:15:43.714226 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.697105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.0.97.2:42327)
0928 05:15:43.697303 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.697307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.697325 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:15:43.697508 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.697513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.697579 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.697661 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:43.699520 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.700276 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.702353 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.702969 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.703065 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.708395 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:43.708411 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:43.708433 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:43.708470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:43.711452 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.712154 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.712161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.712163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.712286 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.712673 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.712679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.712682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.713053 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:43.713358 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.713655 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.713867 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:15:43.714905 20052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:43.855994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19929
I20260928 05:15:43.874019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20063
Sep 28 05:15:43 dist-test-slave-6718 krb5kdc[19689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 05:15:43.891184 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:43.874820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.97.3:44719)
0928 05:15:43.875019 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:43.875025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:43.875136 (+ 111us) server_negotiation.cc:407] Connection header received
0928 05:15:43.875343 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:43.875349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:43.875433 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:43.875559 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:43.876783 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.877617 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.879097 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:43.879635 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:43.879772 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:43.884628 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:43.884643 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:43.884659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:43.884696 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:43.888033 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.888836 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.888853 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.888855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.889030 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:43.889429 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:43.889434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:43.889437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:43.889804 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:43.890146 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:43.890473 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 05:15:43.890851 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 05:15:43.891906 20186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:44.013324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20063
I20260928 05:15:44.033385 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 19702
I20260928 05:15:44.034514 19764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:44.147116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 19702
2026-09-28T05:15:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4523 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T05:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:44Z Disabled control of system clock
I20260928 05:15:44.224658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43761
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:15:44.554854 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:44.555138 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:44.555224 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:44.562383 20242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:44.562508 20242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:44.562592 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:44.562670 20242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:44.562716 20242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:44.572082 20242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:44.573360 20242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:44.575160 20242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:44.586988 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:44.586903 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:44.587013 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:44.588115 20242 server_base.cc:1034] running on GCE node
I20260928 05:15:44.588910 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:44.590343 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:44.591553 20242 hybrid_clock.cc:648] HybridClock initialized: now 1790572544591488 us; error 67 us; skew 500 ppm
I20260928 05:15:44.591910 20242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:44.594300 20242 webserver.cc:460] Webserver started at http://127.0.97.62:35461/ using document root <none> and password file <none>
I20260928 05:15:44.595070 20242 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:44.595160 20242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:44.595479 20242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:44.598464 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "ad35a48cceac4ddc9568b334f7d01729"
format_stamp: "Formatted at 2026-09-28 05:15:44 on dist-test-slave-6718"
server_key: "566355deca9e43eb4ff4210263aabd0c"
server_key_iv: "5ff4f3ca64176e4901e50b24bf0340cd"
server_key_version: "encryptionkey@0"
I20260928 05:15:44.599406 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad35a48cceac4ddc9568b334f7d01729"
format_stamp: "Formatted at 2026-09-28 05:15:44 on dist-test-slave-6718"
server_key: "566355deca9e43eb4ff4210263aabd0c"
server_key_iv: "5ff4f3ca64176e4901e50b24bf0340cd"
server_key_version: "encryptionkey@0"
I20260928 05:15:44.604667 20242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:15:44.608234 20256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:44.609586 20242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:15:44.609766 20242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "ad35a48cceac4ddc9568b334f7d01729"
format_stamp: "Formatted at 2026-09-28 05:15:44 on dist-test-slave-6718"
server_key: "566355deca9e43eb4ff4210263aabd0c"
server_key_iv: "5ff4f3ca64176e4901e50b24bf0340cd"
server_key_version: "encryptionkey@0"
I20260928 05:15:44.609987 20242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:44.629832 20242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:44.630743 20242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:44.631026 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:44.650485 20242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43761
I20260928 05:15:44.650597 20307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43761 every 8 connection(s)
I20260928 05:15:44.651932 20242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:44.655922 20308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:44.657507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20242
I20260928 05:15:44.657693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:44.658257 388 external_mini_cluster.cc:1428] Setting key 7c497ff4e0b469c165de0b2849809726
I20260928 05:15:44.670064 20308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729: Bootstrap starting.
I20260928 05:15:44.671859 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:44.660466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.97.62:43761)
0928 05:15:44.661576 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:44.661587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:44.661628 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:15:44.663330 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:44.663352 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:44.663813 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:44.664220 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:44.664762 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:44.666003 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:44.667408 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:44.668112 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:44.668309 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:44.669485 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:44.669493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:44.669507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:44.670071 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:44.670171 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:44.670192 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:44.670327 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:44.670618 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 05:15:44.670966 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":443,"threads_started":1}
I20260928 05:15:44.673910 20308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:44.675171 20308 log.cc:824] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:44.677892 20308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729: No bootstrap required, opened a new log
I20260928 05:15:44.685772 20308 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad35a48cceac4ddc9568b334f7d01729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43761 } }
I20260928 05:15:44.686134 20308 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:44.686213 20308 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad35a48cceac4ddc9568b334f7d01729, State: Initialized, Role: FOLLOWER
I20260928 05:15:44.686779 20308 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad35a48cceac4ddc9568b334f7d01729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43761 } }
I20260928 05:15:44.687001 20308 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:44.687110 20308 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:44.687242 20308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:44.689713 20308 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad35a48cceac4ddc9568b334f7d01729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43761 } }
I20260928 05:15:44.690197 20308 leader_election.cc:302] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad35a48cceac4ddc9568b334f7d01729; no voters:
I20260928 05:15:44.690765 20308 leader_election.cc:288] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:44.690932 20313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:44.691701 20313 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [term 1 LEADER]: Becoming Leader. State: Replica: ad35a48cceac4ddc9568b334f7d01729, State: Running, Role: LEADER
I20260928 05:15:44.692214 20313 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad35a48cceac4ddc9568b334f7d01729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43761 } }
I20260928 05:15:44.692987 20308 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:44.694779 20315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad35a48cceac4ddc9568b334f7d01729. Latest consensus state: current_term: 1 leader_uuid: "ad35a48cceac4ddc9568b334f7d01729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad35a48cceac4ddc9568b334f7d01729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43761 } } }
I20260928 05:15:44.695176 20315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:44.696707 20314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad35a48cceac4ddc9568b334f7d01729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad35a48cceac4ddc9568b334f7d01729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43761 } } }
I20260928 05:15:44.697089 20314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:44.699478 20318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:44.704712 20318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:44.714848 20318 catalog_manager.cc:1347] Generated new cluster ID: 12d13f934dfe40f8a94e0c27626dba81
I20260928 05:15:44.715021 20318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:44.743651 20318 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:44.745150 20318 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:44.755246 20318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729: Generated new TSK 0
I20260928 05:15:44.756335 20318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:44.774731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43761
--builtin_ntp_servers=127.0.97.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 05:15:45.118854 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:45.119155 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:45.119309 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:45.127456 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:45.127569 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:45.127748 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 05:15:45.138342 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:45.139721 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:45.141659 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:45.155876 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:45.155763 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:45.155936 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:45.156466 20332 server_base.cc:1034] running on GCE node
I20260928 05:15:45.157011 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:45.157889 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:45.159133 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790572545159061 us; error 63 us; skew 500 ppm
I20260928 05:15:45.159482 20332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:45.161835 20332 webserver.cc:460] Webserver started at http://127.0.97.1:35469/ using document root <none> and password file <none>
I20260928 05:15:45.162604 20332 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:45.162737 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:45.163070 20332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:45.165911 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data/instance:
uuid: "761db5355ec049738086ed7fbbdd81b7"
format_stamp: "Formatted at 2026-09-28 05:15:45 on dist-test-slave-6718"
server_key: "35fd8b5d530f0cdf4a63ddbf0e6f97d4"
server_key_iv: "5bca63c415f531060e4e48fa0a22a141"
server_key_version: "encryptionkey@0"
I20260928 05:15:45.166810 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal/instance:
uuid: "761db5355ec049738086ed7fbbdd81b7"
format_stamp: "Formatted at 2026-09-28 05:15:45 on dist-test-slave-6718"
server_key: "35fd8b5d530f0cdf4a63ddbf0e6f97d4"
server_key_iv: "5bca63c415f531060e4e48fa0a22a141"
server_key_version: "encryptionkey@0"
I20260928 05:15:45.172200 20332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:15:45.175824 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:45.177330 20332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:45.177510 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal
uuid: "761db5355ec049738086ed7fbbdd81b7"
format_stamp: "Formatted at 2026-09-28 05:15:45 on dist-test-slave-6718"
server_key: "35fd8b5d530f0cdf4a63ddbf0e6f97d4"
server_key_iv: "5bca63c415f531060e4e48fa0a22a141"
server_key_version: "encryptionkey@0"
I20260928 05:15:45.177732 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:45.198957 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:45.199754 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:45.200033 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:45.200977 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:45.202518 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:45.202692 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:45.202819 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:45.202905 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:45.235625 20332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44817
I20260928 05:15:45.235716 20458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44817 every 8 connection(s)
I20260928 05:15:45.236912 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/data/info.pb
I20260928 05:15:45.243158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20332
I20260928 05:15:45.243356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-0/wal/instance
I20260928 05:15:45.243829 388 external_mini_cluster.cc:1428] Setting key 1fd7a177792526f56049f7952445bdfe
I20260928 05:15:45.246627 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43761
--builtin_ntp_servers=127.0.97.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:15:45.250715 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:45.239756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58941 (local address 127.0.97.62:43761)
0928 05:15:45.240201 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:45.240206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:45.241351 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 05:15:45.243226 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:45.243233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:45.243321 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:45.243448 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:45.244635 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.245608 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.247306 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.248177 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.248341 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:45.249690 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:45.249700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:45.249708 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:45.249735 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:45.249792 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:45.249811 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:45.249906 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:45.250111 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 05:15:45.250227 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
I20260928 05:15:45.251470 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:45.240284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43761 (local address 127.0.97.1:58941)
0928 05:15:45.241191 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:45.241232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:45.242955 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:45.243676 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:45.243684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:45.243769 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:45.244422 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:45.244455 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.245771 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.245774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:45.247158 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:45.247170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.248391 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.248395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:45.248606 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:45.249344 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:45.249347 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:45.249384 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:45.249485 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:45.249914 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:45.249922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:45.249928 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 05:15:45.250062 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 05:15:45.250348 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":278,"threads_started":1}
I20260928 05:15:45.253087 20459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43761
I20260928 05:15:45.253438 20459 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:45.254420 20459 heartbeater.cc:511] Master 127.0.97.62:43761 requested a full tablet report, sending...
I20260928 05:15:45.256671 20273 ts_manager.cc:194] Registered new tserver with Master: 761db5355ec049738086ed7fbbdd81b7 (127.0.97.1:44817)
I20260928 05:15:45.258273 20273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:58941
W20260928 05:15:45.583536 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:45.583806 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:45.583935 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:45.591167 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:45.591280 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:45.591440 20463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 05:15:45.601104 20463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:45.602418 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:45.604225 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:45.617549 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:45.617616 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:45.617586 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:45.618301 20463 server_base.cc:1034] running on GCE node
I20260928 05:15:45.618932 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:45.619904 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:45.621110 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790572545621056 us; error 48 us; skew 500 ppm
I20260928 05:15:45.621429 20463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:45.623811 20463 webserver.cc:460] Webserver started at http://127.0.97.2:39665/ using document root <none> and password file <none>
I20260928 05:15:45.624495 20463 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:45.624608 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:45.624928 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:45.627864 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data/instance:
uuid: "50c418504e844022a3579af5ef60cfea"
format_stamp: "Formatted at 2026-09-28 05:15:45 on dist-test-slave-6718"
server_key: "72ce406752a036c2d2aa1ce8f92182dc"
server_key_iv: "67ad2cbcf73c19c1951c21ff57c88d5a"
server_key_version: "encryptionkey@0"
I20260928 05:15:45.628656 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal/instance:
uuid: "50c418504e844022a3579af5ef60cfea"
format_stamp: "Formatted at 2026-09-28 05:15:45 on dist-test-slave-6718"
server_key: "72ce406752a036c2d2aa1ce8f92182dc"
server_key_iv: "67ad2cbcf73c19c1951c21ff57c88d5a"
server_key_version: "encryptionkey@0"
I20260928 05:15:45.633607 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 05:15:45.637069 20477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:45.638460 20463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:15:45.638650 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal
uuid: "50c418504e844022a3579af5ef60cfea"
format_stamp: "Formatted at 2026-09-28 05:15:45 on dist-test-slave-6718"
server_key: "72ce406752a036c2d2aa1ce8f92182dc"
server_key_iv: "67ad2cbcf73c19c1951c21ff57c88d5a"
server_key_version: "encryptionkey@0"
I20260928 05:15:45.638833 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:45.658309 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:45.659271 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:45.659583 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:45.660743 20463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:45.662431 20463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:45.662549 20463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:45.662647 20463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:45.662729 20463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:45.697196 20463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37287
I20260928 05:15:45.697266 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37287 every 8 connection(s)
I20260928 05:15:45.698572 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/data/info.pb
I20260928 05:15:45.704222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20463
I20260928 05:15:45.704440 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal/instance
I20260928 05:15:45.704967 388 external_mini_cluster.cc:1428] Setting key 58e46a4d788a1ce8f88036c2d30ba8f6
I20260928 05:15:45.708225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43761
--builtin_ntp_servers=127.0.97.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 05:15:45.713418 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:45.701420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42877 (local address 127.0.97.62:43761)
0928 05:15:45.701601 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:45.701605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:45.703101 (+ 1496us) server_negotiation.cc:407] Connection header received
0928 05:15:45.704920 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:45.704926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:45.705026 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:45.705151 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:45.706755 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.707554 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.710391 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.710850 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.710948 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:45.712708 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:45.712712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:45.712715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:45.712732 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:45.712762 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:45.712774 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:45.712843 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:45.713124 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 05:15:45.713201 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:15:45.715180 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:45.701909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43761 (local address 127.0.97.2:42877)
0928 05:15:45.702936 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:45.702980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:45.704649 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:45.705633 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:45.705642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:45.705741 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:45.706575 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:45.706604 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.707714 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.707721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:45.710238 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:45.710258 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:45.711097 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:45.711107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:45.711472 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:45.712271 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:45.712278 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:45.712335 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:45.712467 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:45.712878 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:45.712892 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:45.712904 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 05:15:45.713111 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 05:15:45.713590 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":312,"threads_started":1}
I20260928 05:15:45.716722 20590 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43761
I20260928 05:15:45.717032 20590 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:45.717742 20590 heartbeater.cc:511] Master 127.0.97.62:43761 requested a full tablet report, sending...
I20260928 05:15:45.719215 20273 ts_manager.cc:194] Registered new tserver with Master: 50c418504e844022a3579af5ef60cfea (127.0.97.2:37287)
I20260928 05:15:45.719941 20273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42877
W20260928 05:15:46.051185 20593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:46.051523 20593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:46.051682 20593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:46.058964 20593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:46.059098 20593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:46.059264 20593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 05:15:46.068302 20593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:46.069679 20593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:46.071481 20593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:46.084071 20598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:46.084216 20599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:46.084749 20601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:46.085151 20593 server_base.cc:1034] running on GCE node
I20260928 05:15:46.085788 20593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:46.086797 20593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:46.088037 20593 hybrid_clock.cc:648] HybridClock initialized: now 1790572546087947 us; error 81 us; skew 500 ppm
I20260928 05:15:46.088352 20593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 05:15:46.090598 20593 webserver.cc:460] Webserver started at http://127.0.97.3:32833/ using document root <none> and password file <none>
I20260928 05:15:46.091303 20593 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:46.091430 20593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:46.091741 20593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:46.094520 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data/instance:
uuid: "6c1bf1ef5671443bb35f24be52ad6f6f"
format_stamp: "Formatted at 2026-09-28 05:15:46 on dist-test-slave-6718"
server_key: "ebfd3c590650990c9d576bc393ce1f7b"
server_key_iv: "b0cb9280c5960c707a383a7b09c86f92"
server_key_version: "encryptionkey@0"
I20260928 05:15:46.095371 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6c1bf1ef5671443bb35f24be52ad6f6f"
format_stamp: "Formatted at 2026-09-28 05:15:46 on dist-test-slave-6718"
server_key: "ebfd3c590650990c9d576bc393ce1f7b"
server_key_iv: "b0cb9280c5960c707a383a7b09c86f92"
server_key_version: "encryptionkey@0"
I20260928 05:15:46.100545 20593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:15:46.104017 20607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:46.105301 20593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:46.105476 20593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal
uuid: "6c1bf1ef5671443bb35f24be52ad6f6f"
format_stamp: "Formatted at 2026-09-28 05:15:46 on dist-test-slave-6718"
server_key: "ebfd3c590650990c9d576bc393ce1f7b"
server_key_iv: "b0cb9280c5960c707a383a7b09c86f92"
server_key_version: "encryptionkey@0"
I20260928 05:15:46.105691 20593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:46.119521 20593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:46.120371 20593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:46.120678 20593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:46.121675 20593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 05:15:46.123253 20593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 05:15:46.123374 20593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:46.123510 20593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 05:15:46.123595 20593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:46.157315 20593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36245
I20260928 05:15:46.157366 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36245 every 8 connection(s)
I20260928 05:15:46.158723 20593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/data/info.pb
I20260928 05:15:46.168437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20593
I20260928 05:15:46.168651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal/instance
I20260928 05:15:46.169186 388 external_mini_cluster.cc:1428] Setting key c1d716732c7ab326b77d41e9b9e43551
I20260928 05:15:46.171309 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.161260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46911 (local address 127.0.97.62:43761)
0928 05:15:46.161471 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.161478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.162851 (+ 1373us) server_negotiation.cc:407] Connection header received
0928 05:15:46.164628 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.164632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.164699 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.164787 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:46.165891 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.166684 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.168195 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.168825 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.168978 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.170453 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:46.170458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:46.170461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:46.170481 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:46.170512 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:46.170525 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:46.170653 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.170883 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.170983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:15:46.172303 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.161751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43761 (local address 127.0.97.3:46911)
0928 05:15:46.162693 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.162736 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:46.164350 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:46.165057 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:46.165066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:46.165174 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:15:46.165733 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.165750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.166816 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.166820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.168053 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.168064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.169046 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.169051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.169289 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.170057 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 05:15:46.170060 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:15:46.170118 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 05:15:46.170225 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:15:46.170652 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:15:46.170663 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:15:46.170674 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 05:15:46.170866 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 05:15:46.171262 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":274,"threads_started":1}
I20260928 05:15:46.173599 20720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43761
I20260928 05:15:46.173911 20720 heartbeater.cc:464] Registering TS with master...
I20260928 05:15:46.174706 20720 heartbeater.cc:511] Master 127.0.97.62:43761 requested a full tablet report, sending...
I20260928 05:15:46.176051 20273 ts_manager.cc:194] Registered new tserver with Master: 6c1bf1ef5671443bb35f24be52ad6f6f (127.0.97.3:36245)
I20260928 05:15:46.176856 20273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46911
I20260928 05:15:46.184506 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 05:15:46.194233 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.188543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.97.62:43761)
0928 05:15:46.188776 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.188783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.189077 (+ 294us) server_negotiation.cc:407] Connection header received
0928 05:15:46.189239 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.189243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.189322 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.189408 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:46.189946 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.190756 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.192088 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.192495 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.192585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.193519 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:46.193523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:46.193527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:46.193549 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:46.193581 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:46.193593 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:46.193667 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.193830 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.193900 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:15:46.197980 20273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 05:15:46.204998 20273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 05:15:46.227198 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.217368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37287 (local address 127.0.0.1:43938)
0928 05:15:46.219097 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.219139 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:46.219317 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:46.219959 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:46.219966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:46.219995 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:46.220375 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.220402 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.222326 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.222331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.224820 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.224832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.226201 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.226207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.226419 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.226475 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 05:15:46.226639 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 05:15:46.226821 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"thread_start_us":3948,"threads_started":1}
I20260928 05:15:46.227200 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.217562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.97.2:37287)
0928 05:15:46.218674 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.218693 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.219462 (+ 769us) server_negotiation.cc:407] Connection header received
0928 05:15:46.219563 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.219569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.219753 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.219945 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:46.220598 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.222165 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.225023 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.226000 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.226158 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.226417 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.226681 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.226844 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":802,"threads_started":1}
I20260928 05:15:46.229884 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.218080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36245 (local address 127.0.0.1:60758)
0928 05:15:46.221566 (+ 3486us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.221586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:46.221690 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:46.222781 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:46.222788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:46.222826 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:46.223373 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.223384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.225245 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.225250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.227413 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.227428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.229124 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.229130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.229289 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.229330 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 05:15:46.229435 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:15:46.229554 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3364,"mutex_wait_us":4,"thread_start_us":723,"threads_started":1}
I20260928 05:15:46.229981 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.218791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.0.97.3:36245)
0928 05:15:46.222157 (+ 3366us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.222165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.222199 (+ 34us) server_negotiation.cc:407] Connection header received
0928 05:15:46.222296 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.222303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.222592 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.222783 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:46.223543 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.225119 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.227601 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.228937 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.229073 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.229338 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.229478 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.229620 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3201,"thread_start_us":3064,"threads_started":1}
I20260928 05:15:46.231817 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.222671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:33030)
0928 05:15:46.223444 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.223461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:46.223570 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:46.224333 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:46.224338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:46.224366 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:46.224792 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.224802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.226755 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.226761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.228918 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.228933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.230958 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.230966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.231218 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.231250 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 05:15:46.231343 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 05:15:46.231478 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":58,"thread_start_us":471,"threads_started":1}
I20260928 05:15:46.231901 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.222071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.0.97.1:44817)
0928 05:15:46.222630 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.222638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.223578 (+ 940us) server_negotiation.cc:407] Connection header received
0928 05:15:46.223716 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.223722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.223951 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.224122 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:46.224992 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.226591 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.229128 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.230702 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.230870 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.231126 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.231402 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.231570 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":300,"threads_started":1}
I20260928 05:15:46.232599 20525 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8d555840bb4cf79298caa8c95d1f62 (DEFAULT_TABLE table=test-table [id=92b1b6cd637f438dbd85998abdaa85b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:46.233964 20525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d555840bb4cf79298caa8c95d1f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:46.233951 20655 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8d555840bb4cf79298caa8c95d1f62 (DEFAULT_TABLE table=test-table [id=92b1b6cd637f438dbd85998abdaa85b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:46.235203 20655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d555840bb4cf79298caa8c95d1f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:46.242465 20394 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8d555840bb4cf79298caa8c95d1f62 (DEFAULT_TABLE table=test-table [id=92b1b6cd637f438dbd85998abdaa85b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 05:15:46.244015 20394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d555840bb4cf79298caa8c95d1f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:46.247207 20738 tablet_bootstrap.cc:492] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Bootstrap starting.
I20260928 05:15:46.250216 20738 tablet_bootstrap.cc:654] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:46.251260 20739 tablet_bootstrap.cc:492] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Bootstrap starting.
I20260928 05:15:46.251542 20738 log.cc:824] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:46.254662 20739 tablet_bootstrap.cc:654] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:46.254698 20738 tablet_bootstrap.cc:492] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: No bootstrap required, opened a new log
I20260928 05:15:46.254949 20738 ts_tablet_manager.cc:1397] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 05:15:46.255970 20739 log.cc:824] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:46.258865 20739 tablet_bootstrap.cc:492] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: No bootstrap required, opened a new log
I20260928 05:15:46.259088 20742 tablet_bootstrap.cc:492] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: Bootstrap starting.
I20260928 05:15:46.259238 20739 ts_tablet_manager.cc:1397] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 05:15:46.261212 20459 heartbeater.cc:503] Master 127.0.97.62:43761 was elected leader, sending a full tablet report...
I20260928 05:15:46.262214 20742 tablet_bootstrap.cc:654] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:46.263433 20742 log.cc:824] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:46.265317 20738 raft_consensus.cc:348] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.265816 20738 raft_consensus.cc:374] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:46.265959 20738 raft_consensus.cc:729] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1bf1ef5671443bb35f24be52ad6f6f, State: Initialized, Role: FOLLOWER
I20260928 05:15:46.266685 20738 consensus_queue.cc:260] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.267982 20720 heartbeater.cc:503] Master 127.0.97.62:43761 was elected leader, sending a full tablet report...
I20260928 05:15:46.268615 20738 ts_tablet_manager.cc:1428] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260928 05:15:46.269987 20739 raft_consensus.cc:348] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.270438 20739 raft_consensus.cc:374] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:46.270710 20739 raft_consensus.cc:729] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c418504e844022a3579af5ef60cfea, State: Initialized, Role: FOLLOWER
I20260928 05:15:46.271363 20739 consensus_queue.cc:260] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.272046 20742 tablet_bootstrap.cc:492] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: No bootstrap required, opened a new log
I20260928 05:15:46.272377 20742 ts_tablet_manager.cc:1397] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260928 05:15:46.272455 20590 heartbeater.cc:503] Master 127.0.97.62:43761 was elected leader, sending a full tablet report...
I20260928 05:15:46.273203 20739 ts_tablet_manager.cc:1428] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 05:15:46.280753 20742 raft_consensus.cc:348] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.281195 20742 raft_consensus.cc:374] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:46.281299 20742 raft_consensus.cc:729] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761db5355ec049738086ed7fbbdd81b7, State: Initialized, Role: FOLLOWER
I20260928 05:15:46.281828 20742 consensus_queue.cc:260] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.283562 20742 ts_tablet_manager.cc:1428] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260928 05:15:46.292517 20744 raft_consensus.cc:482] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 05:15:46.292804 20744 raft_consensus.cc:504] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.294627 20744 leader_election.cc:288] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50c418504e844022a3579af5ef60cfea (127.0.97.2:37287), 761db5355ec049738086ed7fbbdd81b7 (127.0.97.1:44817)
I20260928 05:15:46.301404 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.294980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50589 (local address 127.0.97.2:37287)
0928 05:15:46.295180 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.295188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.295211 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:15:46.295446 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.295452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.295552 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.295639 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:46.296282 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.297280 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.299331 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.300599 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.300716 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.300807 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.301119 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.301211 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:15:46.301524 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.294898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37287 (local address 127.0.97.3:50589)
0928 05:15:46.295096 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.295117 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:46.295243 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:46.295881 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:46.295885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:46.295910 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:46.296124 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.296134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.297405 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.297410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.299169 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.299186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.300761 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.300767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.300956 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.300989 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 05:15:46.301090 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 05:15:46.301228 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:15:46.302397 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.295571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56243 (local address 127.0.97.1:44817)
0928 05:15:46.295816 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.295825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.296277 (+ 452us) server_negotiation.cc:407] Connection header received
0928 05:15:46.296430 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.296433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.296496 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.296562 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 05:15:46.297131 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.298128 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.300468 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.301640 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.301770 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.301870 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.302101 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.302221 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 05:15:46.302489 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.295299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.97.3:56243)
0928 05:15:46.296146 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.296164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:15:46.296302 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:15:46.296602 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:15:46.296607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:15:46.296632 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 05:15:46.296990 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.297001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.298325 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.298329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.300276 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:15:46.300291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.301798 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.301804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:15:46.301943 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.301966 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 05:15:46.302043 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 05:15:46.302154 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":490,"threads_started":1}
I20260928 05:15:46.302748 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d555840bb4cf79298caa8c95d1f62" candidate_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c418504e844022a3579af5ef60cfea" is_pre_election: true
I20260928 05:15:46.303385 20545 raft_consensus.cc:2454] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1bf1ef5671443bb35f24be52ad6f6f in term 0.
I20260928 05:15:46.303665 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d555840bb4cf79298caa8c95d1f62" candidate_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761db5355ec049738086ed7fbbdd81b7" is_pre_election: true
I20260928 05:15:46.304090 20610 leader_election.cc:302] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c418504e844022a3579af5ef60cfea, 6c1bf1ef5671443bb35f24be52ad6f6f; no voters:
I20260928 05:15:46.304200 20414 raft_consensus.cc:2454] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1bf1ef5671443bb35f24be52ad6f6f in term 0.
I20260928 05:15:46.304549 20744 raft_consensus.cc:2781] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 05:15:46.304750 20744 raft_consensus.cc:482] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 05:15:46.304875 20744 raft_consensus.cc:3037] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:46.307178 20744 raft_consensus.cc:504] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.307752 20744 leader_election.cc:288] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [CANDIDATE]: Term 1 election: Requested vote from peers 50c418504e844022a3579af5ef60cfea (127.0.97.2:37287), 761db5355ec049738086ed7fbbdd81b7 (127.0.97.1:44817)
I20260928 05:15:46.308254 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d555840bb4cf79298caa8c95d1f62" candidate_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c418504e844022a3579af5ef60cfea"
I20260928 05:15:46.308322 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d555840bb4cf79298caa8c95d1f62" candidate_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761db5355ec049738086ed7fbbdd81b7"
I20260928 05:15:46.308475 20545 raft_consensus.cc:3037] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:46.308508 20414 raft_consensus.cc:3037] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:46.311789 20414 raft_consensus.cc:2454] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1bf1ef5671443bb35f24be52ad6f6f in term 1.
I20260928 05:15:46.311789 20545 raft_consensus.cc:2454] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1bf1ef5671443bb35f24be52ad6f6f in term 1.
I20260928 05:15:46.312181 20610 leader_election.cc:302] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1bf1ef5671443bb35f24be52ad6f6f, 761db5355ec049738086ed7fbbdd81b7; no voters:
I20260928 05:15:46.312474 20744 raft_consensus.cc:2781] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:46.312933 20744 raft_consensus.cc:686] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 1 LEADER]: Becoming Leader. State: Replica: 6c1bf1ef5671443bb35f24be52ad6f6f, State: Running, Role: LEADER
I20260928 05:15:46.313439 20744 consensus_queue.cc:237] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } }
I20260928 05:15:46.318264 20273 catalog_manager.cc:5495] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1bf1ef5671443bb35f24be52ad6f6f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1bf1ef5671443bb35f24be52ad6f6f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36245 } health_report { overall_health: HEALTHY } } }
I20260928 05:15:46.351717 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.346433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.0.97.3:36245)
0928 05:15:46.346655 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.346661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.346680 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:15:46.346902 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.346906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.347003 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.347164 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:15:46.347517 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.348245 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.349633 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.350048 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.350174 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.350686 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:15:46.351166 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:15:46.351328 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.351383 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.351469 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:15:46.359789 20545 raft_consensus.cc:1261] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 1 FOLLOWER]: Refusing update from remote peer 6c1bf1ef5671443bb35f24be52ad6f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:46.359789 20414 raft_consensus.cc:1261] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 1 FOLLOWER]: Refusing update from remote peer 6c1bf1ef5671443bb35f24be52ad6f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:15:46.360788 20748 consensus_queue.cc:1035] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "761db5355ec049738086ed7fbbdd81b7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:46.361135 20744 consensus_queue.cc:1035] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c418504e844022a3579af5ef60cfea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:15:46.372575 20751 mvcc.cc:205] Tried to move back new op lower bound from 7334185149879058432 to 7334185149703921664. Current Snapshot: MvccSnapshot[applied={T|T < 7334185149879058432}]
I20260928 05:15:46.374374 20753 mvcc.cc:205] Tried to move back new op lower bound from 7334185149879058432 to 7334185149703921664. Current Snapshot: MvccSnapshot[applied={T|T < 7334185149879058432}]
I20260928 05:15:46.385775 20273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:46.386078 20273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 05:15:46.390003 20273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad35a48cceac4ddc9568b334f7d01729: Sending DeleteTablet for 3 replicas of tablet 3f8d555840bb4cf79298caa8c95d1f62
I20260928 05:15:46.391335 20655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8d555840bb4cf79298caa8c95d1f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:46 UTC) from {username='slave'} at 127.0.0.1:60758
I20260928 05:15:46.391469 20525 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8d555840bb4cf79298caa8c95d1f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:46 UTC) from {username='slave'} at 127.0.0.1:43938
I20260928 05:15:46.391680 20394 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8d555840bb4cf79298caa8c95d1f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 05:15:46 UTC) from {username='slave'} at 127.0.0.1:33030
I20260928 05:15:46.392390 20761 tablet_replica.cc:330] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: stopping tablet replica
I20260928 05:15:46.392940 20761 raft_consensus.cc:2229] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 1 LEADER]: Raft consensus shutting down.
I20260928 05:15:46.393008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20332
I20260928 05:15:46.393759 20761 raft_consensus.cc:2258] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:46.401199 20761 ts_tablet_manager.cc:1905] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:46.407141 20763 tablet_replica.cc:330] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: stopping tablet replica
I20260928 05:15:46.407719 20763 raft_consensus.cc:2229] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:46.408377 20763 raft_consensus.cc:2258] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:46.411341 20763 ts_tablet_manager.cc:1905] T 3f8d555840bb4cf79298caa8c95d1f62 P 761db5355ec049738086ed7fbbdd81b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:46.412061 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.395185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.97.1:44817)
0928 05:15:46.395369 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.395376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.395624 (+ 248us) server_negotiation.cc:407] Connection header received
0928 05:15:46.395906 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.395925 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.396034 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.396122 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:46.396960 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.398700 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.401157 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.401848 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.401967 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.409772 (+ 7805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:46.409790 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:46.409819 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:46.410695 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:46.410763 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:46.410792 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:46.410923 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.411553 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.411673 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:15:46.412757 20761 ts_tablet_manager.cc:1918] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:46.412981 20761 log.cc:1196] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-2/wal/wals/3f8d555840bb4cf79298caa8c95d1f62
I20260928 05:15:46.412993 20762 tablet_replica.cc:330] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: stopping tablet replica
I20260928 05:15:46.413319 20454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:46.413436 20762 raft_consensus.cc:2229] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 05:15:46.413753 20761 ts_tablet_manager.cc:1939] T 3f8d555840bb4cf79298caa8c95d1f62 P 6c1bf1ef5671443bb35f24be52ad6f6f: Deleting consensus metadata
I20260928 05:15:46.413998 20762 raft_consensus.cc:2258] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 05:15:46.416004 20762 ts_tablet_manager.cc:1905] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 05:15:46.416602 20257 catalog_manager.cc:4841] TS 6c1bf1ef5671443bb35f24be52ad6f6f (127.0.97.3:36245): tablet 3f8d555840bb4cf79298caa8c95d1f62 (table test-table [id=92b1b6cd637f438dbd85998abdaa85b3]) successfully deleted
I20260928 05:15:46.423090 20762 ts_tablet_manager.cc:1918] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 05:15:46.423271 20762 log.cc:1196] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Deleting WAL directory at /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790572228671169-388-0/minicluster-data/ts-1/wal/wals/3f8d555840bb4cf79298caa8c95d1f62
I20260928 05:15:46.423832 20762 ts_tablet_manager.cc:1939] T 3f8d555840bb4cf79298caa8c95d1f62 P 50c418504e844022a3579af5ef60cfea: Deleting consensus metadata
I20260928 05:15:46.425743 20259 catalog_manager.cc:4841] TS 50c418504e844022a3579af5ef60cfea (127.0.97.2:37287): tablet 3f8d555840bb4cf79298caa8c95d1f62 (table test-table [id=92b1b6cd637f438dbd85998abdaa85b3]) successfully deleted
I20260928 05:15:46.514202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20332
I20260928 05:15:46.539709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20463
W20260928 05:15:46.539757 20259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44817 (error 108)
I20260928 05:15:46.542114 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.541648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40676)
0928 05:15:46.541806 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.541928 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 05:15:46.542423 20259 catalog_manager.cc:4584] TS 761db5355ec049738086ed7fbbdd81b7 (127.0.97.1:44817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f8d555840bb4cf79298caa8c95d1f62: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
I20260928 05:15:46.548105 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.540678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.97.2:37287)
0928 05:15:46.540893 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.540900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.540923 (+ 23us) server_negotiation.cc:407] Connection header received
0928 05:15:46.541133 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.541139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.541255 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.541352 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:46.541981 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.543366 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.544688 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.545336 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.545465 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.546481 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:46.546492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:46.546510 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:46.547299 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:46.547371 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:46.547395 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:46.547506 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.547609 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.547713 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 05:15:46.548789 20585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:46.596102 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.595575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40684)
0928 05:15:46.595801 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.595908 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:15:46.597271 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.596836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40688)
0928 05:15:46.597011 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.597114 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 05:15:46.638942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20463
I20260928 05:15:46.658154 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.657521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40704)
0928 05:15:46.657837 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.657955 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260928 05:15:46.659356 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.658935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40712)
0928 05:15:46.659135 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.659216 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 05:15:46.661317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20593
I20260928 05:15:46.670154 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.662089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.0.97.3:36245)
0928 05:15:46.662266 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:46.662272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:46.662292 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:15:46.662450 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:46.662455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:46.662583 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:46.662703 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:46.663150 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.664386 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.666235 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:46.667163 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:46.667295 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:46.668444 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:46.668455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:46.668473 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:46.669311 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:46.669371 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:46.669397 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:46.669525 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:46.669662 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 05:15:46.669763 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 05:15:46.670920 20715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:46.740624 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.740089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40720)
0928 05:15:46.740298 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.740413 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 05:15:46.741724 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.741297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40728)
0928 05:15:46.741521 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.741615 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 05:15:46.794853 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20593
I20260928 05:15:46.813787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20242
I20260928 05:15:46.815017 20303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:46.922686 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:46.921966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44817 (local address 127.0.0.1:40736)
0928 05:15:46.922318 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 05:15:46.922472 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260928 05:15:46.922749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20242
2026-09-28T05:15:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2774 ms)
[----------] 6 tests from EncryptionPolicyTest (21751 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:47 dist-test-slave-6718 krb5kdc[20772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:47 dist-test-slave-6718 krb5kdc[20772](info): set up 2 sockets
Sep 28 05:15:47 dist-test-slave-6718 krb5kdc[20772](info): commencing operation
krb5kdc: starting...
W20260928 05:15:48.055023 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:48 dist-test-slave-6718 krb5kdc[20772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:48.318120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:48.646981 20785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:48.647375 20785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:48.647567 20785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:48.654804 20785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:48.654953 20785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:48.655063 20785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:48.655169 20785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:48.655248 20785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:48.664983 20785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:48.666320 20785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:48.667629 20785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:48.679242 20791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:48.679322 20790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:48.679945 20793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:48.679992 20785 server_base.cc:1034] running on GCE node
I20260928 05:15:48.680837 20785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:48.682276 20785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:48.683514 20785 hybrid_clock.cc:648] HybridClock initialized: now 1790572548683452 us; error 70 us; skew 500 ppm
I20260928 05:15:48.683863 20785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:48 dist-test-slave-6718 krb5kdc[20772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:48.686923 20785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:15:48.688390 20785 webserver.cc:460] Webserver started at http://127.0.0.1:42779/ using document root <none> and password file <none>
I20260928 05:15:48.689069 20785 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:48.689213 20785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:48.689512 20785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:48.692466 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "16725742a53148bbbb21ba316affd1e3"
format_stamp: "Formatted at 2026-09-28 05:15:48 on dist-test-slave-6718"
server_key: "d185fd9c9865d657e825830c113231c5"
server_key_iv: "05bc421c83116c3eba8613dc083d8a5d"
server_key_version: "encryptionkey@0"
I20260928 05:15:48.693276 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "16725742a53148bbbb21ba316affd1e3"
format_stamp: "Formatted at 2026-09-28 05:15:48 on dist-test-slave-6718"
server_key: "d185fd9c9865d657e825830c113231c5"
server_key_iv: "05bc421c83116c3eba8613dc083d8a5d"
server_key_version: "encryptionkey@0"
I20260928 05:15:48.698351 20785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:15:48.701797 20800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:48.703137 20785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:15:48.703321 20785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "16725742a53148bbbb21ba316affd1e3"
format_stamp: "Formatted at 2026-09-28 05:15:48 on dist-test-slave-6718"
server_key: "d185fd9c9865d657e825830c113231c5"
server_key_iv: "05bc421c83116c3eba8613dc083d8a5d"
server_key_version: "encryptionkey@0"
I20260928 05:15:48.703528 20785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:48.715302 20785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:48.719151 20785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:48.719426 20785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:48.738503 20785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39499
I20260928 05:15:48.738673 20851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39499 every 8 connection(s)
I20260928 05:15:48.740106 20785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:48.741835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20785
I20260928 05:15:48.742053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:48.742743 388 external_mini_cluster.cc:1428] Setting key fbafd7b6b24ffc7dc20fa9263b181bef
I20260928 05:15:48.744590 20852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:15:48 dist-test-slave-6718 krb5kdc[20772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:15:48.759004 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3: Bootstrap starting.
I20260928 05:15:48.762250 20852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:48.763338 20852 log.cc:824] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:48.766700 20852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3: No bootstrap required, opened a new log
I20260928 05:15:48.767174 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:48.744627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32783 (local address 127.0.0.1:39499)
0928 05:15:48.745459 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:48.745470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:48.745513 (+ 43us) server_negotiation.cc:407] Connection header received
0928 05:15:48.747318 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:48.747340 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:48.747760 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:48.748130 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:48.750066 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:48.751560 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:48.753432 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:48.754130 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:48.754277 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:48.759289 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:48.759297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:48.759313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:48.759358 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:48.762663 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:48.763534 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:48.763547 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:48.763553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:48.763723 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:48.764338 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:48.764344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:48.764348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:48.764976 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:48.765213 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:48.765697 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 05:15:48.766010 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":383,"threads_started":1}
I20260928 05:15:48.771358 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:15:48.775017 20852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16725742a53148bbbb21ba316affd1e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260928 05:15:48.775403 20852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:48.775517 20852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16725742a53148bbbb21ba316affd1e3, State: Initialized, Role: FOLLOWER
I20260928 05:15:48.776130 20852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16725742a53148bbbb21ba316affd1e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260928 05:15:48.776389 20852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:48.776546 20852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:48.776734 20852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:48.778926 20852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16725742a53148bbbb21ba316affd1e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260928 05:15:48.779429 20852 leader_election.cc:302] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16725742a53148bbbb21ba316affd1e3; no voters:
I20260928 05:15:48.780012 20852 leader_election.cc:288] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:48.780160 20862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:48.780879 20862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [term 1 LEADER]: Becoming Leader. State: Replica: 16725742a53148bbbb21ba316affd1e3, State: Running, Role: LEADER
I20260928 05:15:48.781476 20862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16725742a53148bbbb21ba316affd1e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260928 05:15:48.784829 20852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:48.786682 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16725742a53148bbbb21ba316affd1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16725742a53148bbbb21ba316affd1e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } } }
I20260928 05:15:48.787056 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:48.792979 20867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:48.793493 20863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16725742a53148bbbb21ba316affd1e3. Latest consensus state: current_term: 1 leader_uuid: "16725742a53148bbbb21ba316affd1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16725742a53148bbbb21ba316affd1e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } } }
I20260928 05:15:48.793823 20863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:48.799775 20867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:48.800367 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:48.776245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44121 (local address 127.0.0.1:39499)
0928 05:15:48.776512 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:48.776520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:48.779110 (+ 2590us) server_negotiation.cc:407] Connection header received
0928 05:15:48.779326 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:48.779332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:48.779411 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:48.779522 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:48.781592 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:48.782791 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:48.785263 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:48.785957 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:48.786122 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:48.790399 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:48.790408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:48.790416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:48.790468 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:48.795355 (+ 4887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:48.796625 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:48.796633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:48.796639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:48.796896 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:48.797647 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:48.797655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:48.797661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:48.798704 (+ 1043us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:48.799166 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:48.799969 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 05:15:48.800120 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 05:15:48.812741 20867 catalog_manager.cc:1347] Generated new cluster ID: d125dcefbd334354a9764e5896dfd4c6
I20260928 05:15:48.812879 20867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:48.822973 20867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:48.824795 20867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:48.831801 20867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16725742a53148bbbb21ba316affd1e3: Generated new TSK 0
I20260928 05:15:48.832861 20867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:48.844698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20785
I20260928 05:15:48.846014 20847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:48.935472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20785
W20260928 05:15:48.953154 20770 connection.cc:535] client connection to 127.0.0.1:39499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39499): Connection reset by peer (error 104)
2026-09-28T05:15:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:49 dist-test-slave-6718 krb5kdc[20885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:49 dist-test-slave-6718 krb5kdc[20885](info): set up 2 sockets
Sep 28 05:15:49 dist-test-slave-6718 krb5kdc[20885](info): commencing operation
krb5kdc: starting...
W20260928 05:15:50.034812 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:50 dist-test-slave-6718 krb5kdc[20885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:50.289193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:50.616039 20898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:50.616324 20898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:50.616499 20898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:50.623785 20898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:50.623906 20898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:50.624015 20898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:50.624125 20898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:50.624208 20898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:50.634145 20898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:50.635466 20898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:50.636770 20898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:50.648175 20903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:50.648231 20904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:50.648370 20906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:50.649217 20898 server_base.cc:1034] running on GCE node
I20260928 05:15:50.650022 20898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:50.651520 20898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:50.652720 20898 hybrid_clock.cc:648] HybridClock initialized: now 1790572550652681 us; error 32 us; skew 500 ppm
I20260928 05:15:50.653064 20898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:50 dist-test-slave-6718 krb5kdc[20885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:50.656179 20898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:15:50.657675 20898 webserver.cc:460] Webserver started at http://127.0.0.1:43573/ using document root <none> and password file <none>
I20260928 05:15:50.658356 20898 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:50.658505 20898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:50.658869 20898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:50.661834 20898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "d9c259e232cf47d0b8080c509e41e1ed"
format_stamp: "Formatted at 2026-09-28 05:15:50 on dist-test-slave-6718"
server_key: "b9471b5c820aa63eff6b638b9b662dd7"
server_key_iv: "f6a1b58a748dfa105980233c55d9c00b"
server_key_version: "encryptionkey@0"
I20260928 05:15:50.662693 20898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "d9c259e232cf47d0b8080c509e41e1ed"
format_stamp: "Formatted at 2026-09-28 05:15:50 on dist-test-slave-6718"
server_key: "b9471b5c820aa63eff6b638b9b662dd7"
server_key_iv: "f6a1b58a748dfa105980233c55d9c00b"
server_key_version: "encryptionkey@0"
I20260928 05:15:50.667649 20898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:15:50.671511 20913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:50.672879 20898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:50.673060 20898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "d9c259e232cf47d0b8080c509e41e1ed"
format_stamp: "Formatted at 2026-09-28 05:15:50 on dist-test-slave-6718"
server_key: "b9471b5c820aa63eff6b638b9b662dd7"
server_key_iv: "f6a1b58a748dfa105980233c55d9c00b"
server_key_version: "encryptionkey@0"
I20260928 05:15:50.673250 20898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:50.692364 20898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:50.696561 20898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:50.696846 20898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:50.715188 20898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36299
I20260928 05:15:50.715234 20964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36299 every 8 connection(s)
I20260928 05:15:50.716581 20898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:50.720266 20965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:50.722970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 20898
I20260928 05:15:50.723150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:50.723663 388 external_mini_cluster.cc:1428] Setting key 936d3176a8208c14d54149a1b14c07fd
I20260928 05:15:50.732525 20965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed: Bootstrap starting.
I20260928 05:15:50.736188 20965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:50.737429 20965 log.cc:824] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:50.737668 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:50.725494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57225 (local address 127.0.0.1:36299)
0928 05:15:50.726353 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:50.726365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:50.726406 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:15:50.728173 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:50.728196 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:50.728596 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:50.728951 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:50.729568 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:50.730980 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:50.733076 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:50.733781 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:50.733915 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:50.735295 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:50.735303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:50.735315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:50.735865 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:50.735938 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:50.735963 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:50.736080 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:50.736443 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 05:15:50.736607 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":418,"threads_started":1}
I20260928 05:15:50.740749 20965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed: No bootstrap required, opened a new log
I20260928 05:15:50.741727 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:15:50.749502 20965 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c259e232cf47d0b8080c509e41e1ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36299 } }
I20260928 05:15:50.749927 20965 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:50.750062 20965 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c259e232cf47d0b8080c509e41e1ed, State: Initialized, Role: FOLLOWER
I20260928 05:15:50.750823 20965 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c259e232cf47d0b8080c509e41e1ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36299 } }
I20260928 05:15:50.751143 20965 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:50.751325 20965 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:50.751559 20965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:50.753238 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:50.745926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49737 (local address 127.0.0.1:36299)
0928 05:15:50.746115 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:50.746121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:50.746651 (+ 530us) server_negotiation.cc:407] Connection header received
0928 05:15:50.746823 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:50.746833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:50.746922 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:50.747032 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:50.747506 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:50.748738 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:50.750328 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:50.751025 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:50.751160 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:50.752444 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:50.752452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:50.752462 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:50.752485 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:50.752561 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:50.752589 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:50.752704 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:50.752847 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 05:15:50.752899 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:15:50.754487 20965 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c259e232cf47d0b8080c509e41e1ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36299 } }
I20260928 05:15:50.755091 20965 leader_election.cc:302] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9c259e232cf47d0b8080c509e41e1ed; no voters:
I20260928 05:15:50.755674 20965 leader_election.cc:288] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:50.755883 20975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:50.756613 20975 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [term 1 LEADER]: Becoming Leader. State: Replica: d9c259e232cf47d0b8080c509e41e1ed, State: Running, Role: LEADER
I20260928 05:15:50.757228 20975 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c259e232cf47d0b8080c509e41e1ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36299 } }
I20260928 05:15:50.757895 20965 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:50.760456 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9c259e232cf47d0b8080c509e41e1ed. Latest consensus state: current_term: 1 leader_uuid: "d9c259e232cf47d0b8080c509e41e1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c259e232cf47d0b8080c509e41e1ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36299 } } }
I20260928 05:15:50.760927 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:50.762427 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9c259e232cf47d0b8080c509e41e1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c259e232cf47d0b8080c509e41e1ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36299 } } }
I20260928 05:15:50.763461 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:50.764686 20984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:50.771196 20984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:50.781266 20984 catalog_manager.cc:1347] Generated new cluster ID: 9126f46dc3f64ffa8c3438e5d20f36e2
I20260928 05:15:50.781390 20984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:50.798820 20984 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:50.800390 20984 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:50.817402 20984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9c259e232cf47d0b8080c509e41e1ed: Generated new TSK 0
I20260928 05:15:50.818280 20984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:50.829627 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20898
W20260928 05:15:50.830834 20960 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57225
W20260928 05:15:50.831645 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 20898: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:15:50.831828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 20898
W20260928 05:15:50.848675 20883 connection.cc:535] client connection to 127.0.0.1:36299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36299): Connection reset by peer (error 104)
2026-09-28T05:15:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:50 dist-test-slave-6718 krb5kdc[20996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:50 dist-test-slave-6718 krb5kdc[20996](info): set up 2 sockets
Sep 28 05:15:50 dist-test-slave-6718 krb5kdc[20996](info): commencing operation
krb5kdc: starting...
W20260928 05:15:51.938998 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:52 dist-test-slave-6718 krb5kdc[20996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:52.193802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:52.524645 21009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:52.524995 21009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:52.525097 21009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:52.532491 21009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:52.532619 21009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:52.532676 21009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:52.532738 21009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:52.532783 21009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:52.542877 21009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:52.544124 21009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:52.545377 21009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:52.558140 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:52.558408 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:52.558212 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:52.564293 21009 server_base.cc:1034] running on GCE node
I20260928 05:15:52.565167 21009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:52.566737 21009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:52.567955 21009 hybrid_clock.cc:648] HybridClock initialized: now 1790572552567914 us; error 31 us; skew 500 ppm
I20260928 05:15:52.568285 21009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:52 dist-test-slave-6718 krb5kdc[20996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:52.575281 21009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:15:52.576975 21009 webserver.cc:460] Webserver started at http://127.0.0.1:36069/ using document root <none> and password file <none>
I20260928 05:15:52.577728 21009 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:52.577863 21009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:52.578203 21009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:52.581342 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "c00488997d0d475c9cef0dd3b95ca8e9"
format_stamp: "Formatted at 2026-09-28 05:15:52 on dist-test-slave-6718"
server_key: "3327eca514aabf6ec3934854008760f0"
server_key_iv: "141522e1ee0afbb27c08c2e14b2847c4"
server_key_version: "encryptionkey@0"
I20260928 05:15:52.582134 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "c00488997d0d475c9cef0dd3b95ca8e9"
format_stamp: "Formatted at 2026-09-28 05:15:52 on dist-test-slave-6718"
server_key: "3327eca514aabf6ec3934854008760f0"
server_key_iv: "141522e1ee0afbb27c08c2e14b2847c4"
server_key_version: "encryptionkey@0"
I20260928 05:15:52.587527 21009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:15:52.590976 21024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:52.592267 21009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:15:52.592454 21009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "c00488997d0d475c9cef0dd3b95ca8e9"
format_stamp: "Formatted at 2026-09-28 05:15:52 on dist-test-slave-6718"
server_key: "3327eca514aabf6ec3934854008760f0"
server_key_iv: "141522e1ee0afbb27c08c2e14b2847c4"
server_key_version: "encryptionkey@0"
I20260928 05:15:52.592650 21009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:52.598508 21009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:52.602779 21009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:52.603088 21009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:52.621810 21009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44741
I20260928 05:15:52.621851 21075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44741 every 8 connection(s)
I20260928 05:15:52.623281 21009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:52.626864 21076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:52.627708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21009
I20260928 05:15:52.627892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:52.628417 388 external_mini_cluster.cc:1428] Setting key 190dc68f3e809544e9b9627e2aad4ada
I20260928 05:15:52.636027 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:52.630180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46751 (local address 127.0.0.1:44741)
0928 05:15:52.631032 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:52.631043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:52.631083 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:15:52.632664 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:52.632685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:52.633026 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:52.633372 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:52.634460 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:52.634466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:52.634479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:52.634492 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:52.634606 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:52.634629 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:52.634821 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:52.634921 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 05:15:52.635121 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":424,"threads_started":1}
I20260928 05:15:52.639855 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:15:52.641214 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9: Bootstrap starting.
I20260928 05:15:52.644687 21076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:52.645730 21076 log.cc:824] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:52.646755 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:52.644189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45091 (local address 127.0.0.1:44741)
0928 05:15:52.644374 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:52.644381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:52.644740 (+ 359us) server_negotiation.cc:407] Connection header received
0928 05:15:52.644961 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:52.644969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:52.645021 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:52.645139 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:52.646073 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:52.646078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:52.646083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:52.646100 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:52.646144 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:52.646165 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:52.646288 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:52.646379 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 05:15:52.646446 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 05:15:52.649061 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9: No bootstrap required, opened a new log
I20260928 05:15:52.656898 21076 raft_consensus.cc:348] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 05:15:52.657418 21076 raft_consensus.cc:374] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:52.657550 21076 raft_consensus.cc:729] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00488997d0d475c9cef0dd3b95ca8e9, State: Initialized, Role: FOLLOWER
I20260928 05:15:52.658141 21076 consensus_queue.cc:260] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 05:15:52.658341 21076 raft_consensus.cc:388] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:52.658442 21076 raft_consensus.cc:482] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:52.658660 21076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:52.660916 21076 raft_consensus.cc:504] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 05:15:52.661375 21076 leader_election.cc:302] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c00488997d0d475c9cef0dd3b95ca8e9; no voters:
I20260928 05:15:52.661970 21076 leader_election.cc:288] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:52.662099 21086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:52.662828 21086 raft_consensus.cc:686] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [term 1 LEADER]: Becoming Leader. State: Replica: c00488997d0d475c9cef0dd3b95ca8e9, State: Running, Role: LEADER
I20260928 05:15:52.663458 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 05:15:52.664327 21076 sys_catalog.cc:564] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:52.666301 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } } }
I20260928 05:15:52.666797 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:52.666327 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c00488997d0d475c9cef0dd3b95ca8e9. Latest consensus state: current_term: 1 leader_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c00488997d0d475c9cef0dd3b95ca8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } } }
I20260928 05:15:52.666890 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:52.668744 21092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:52.673308 21092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:52.681459 21092 catalog_manager.cc:1347] Generated new cluster ID: 18e435fbfef6404f81bfb059317fc759
I20260928 05:15:52.681581 21092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:52.717748 21092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:52.718281 21092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:52.726675 21092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c00488997d0d475c9cef0dd3b95ca8e9: Generated new TSK 0
I20260928 05:15:52.727694 21092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:52.740422 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21009
W20260928 05:15:52.741932 21071 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46751
W20260928 05:15:52.742709 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:15:52.742851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21009
2026-09-28T05:15:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:52 dist-test-slave-6718 krb5kdc[21107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:52 dist-test-slave-6718 krb5kdc[21107](info): set up 2 sockets
Sep 28 05:15:52 dist-test-slave-6718 krb5kdc[21107](info): commencing operation
krb5kdc: starting...
W20260928 05:15:53.839530 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:54 dist-test-slave-6718 krb5kdc[21107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:54.098285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:54.435331 21120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:54.435695 21120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:54.435868 21120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:54.443331 21120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:54.443481 21120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:54.443600 21120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:54.443715 21120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:54.443794 21120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:54.453764 21120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:54.455153 21120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:54.456449 21120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:54.468605 21125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:54.468628 21128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:54.468537 21126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:54.469311 21120 server_base.cc:1034] running on GCE node
I20260928 05:15:54.470147 21120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:54.471633 21120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:54.472831 21120 hybrid_clock.cc:648] HybridClock initialized: now 1790572554472785 us; error 34 us; skew 500 ppm
I20260928 05:15:54.473174 21120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:54 dist-test-slave-6718 krb5kdc[21107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:54.476379 21120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:15:54.477908 21120 webserver.cc:460] Webserver started at http://127.0.0.1:38641/ using document root <none> and password file <none>
I20260928 05:15:54.478650 21120 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:54.478792 21120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:54.479094 21120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:54.482005 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08"
format_stamp: "Formatted at 2026-09-28 05:15:54 on dist-test-slave-6718"
server_key: "81e1bcb9cd663eb9995f87b0a3f109b0"
server_key_iv: "44dba9fac707c86d222fffb3788f2d7a"
server_key_version: "encryptionkey@0"
I20260928 05:15:54.482801 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08"
format_stamp: "Formatted at 2026-09-28 05:15:54 on dist-test-slave-6718"
server_key: "81e1bcb9cd663eb9995f87b0a3f109b0"
server_key_iv: "44dba9fac707c86d222fffb3788f2d7a"
server_key_version: "encryptionkey@0"
I20260928 05:15:54.487975 21120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:15:54.491453 21135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:54.492822 21120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:15:54.493016 21120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08"
format_stamp: "Formatted at 2026-09-28 05:15:54 on dist-test-slave-6718"
server_key: "81e1bcb9cd663eb9995f87b0a3f109b0"
server_key_iv: "44dba9fac707c86d222fffb3788f2d7a"
server_key_version: "encryptionkey@0"
I20260928 05:15:54.493229 21120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:54.508201 21120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:54.512449 21120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:54.512763 21120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:54.531141 21120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46137
I20260928 05:15:54.531173 21186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46137 every 8 connection(s)
I20260928 05:15:54.532619 21120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:54.536401 21187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:54.536512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21120
I20260928 05:15:54.536698 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:54.537216 388 external_mini_cluster.cc:1428] Setting key abcb9693e74c1493b375ad9a89db239a
I20260928 05:15:54.550994 21187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08: Bootstrap starting.
Sep 28 05:15:54 dist-test-slave-6718 krb5kdc[21107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:15:54.554073 21187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:54.555218 21187 log.cc:824] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:54.557917 21187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08: No bootstrap required, opened a new log
I20260928 05:15:54.560345 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:54.539060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50733 (local address 127.0.0.1:46137)
0928 05:15:54.539986 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:54.540000 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:54.540046 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:15:54.541596 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:54.541616 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:54.542031 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:54.542369 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:54.544129 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:54.545441 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:54.547326 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:54.548018 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:54.548166 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:54.552859 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:54.552867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:54.552881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:54.552917 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:54.556269 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:54.557024 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:54.557030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:54.557036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:54.557210 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:54.557614 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:54.557618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:54.557620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:54.558171 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:54.558515 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:54.558929 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 05:15:54.559332 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":495,"threads_started":1}
I20260928 05:15:54.564143 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:15:54.565912 21187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 05:15:54.566273 21187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:54.566377 21187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ed83a428b4fa9bb39c8d1cdfefc08, State: Initialized, Role: FOLLOWER
I20260928 05:15:54.567094 21187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 05:15:54.567330 21187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:54.567474 21187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:54.567638 21187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:54.570245 21187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 05:15:54.570827 21187 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8ed83a428b4fa9bb39c8d1cdfefc08; no voters:
I20260928 05:15:54.571445 21187 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:54.571645 21197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:54.572338 21197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8ed83a428b4fa9bb39c8d1cdfefc08, State: Running, Role: LEADER
I20260928 05:15:54.573071 21197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260928 05:15:54.575215 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8ed83a428b4fa9bb39c8d1cdfefc08. Latest consensus state: current_term: 1 leader_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } } }
I20260928 05:15:54.575594 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:54.579682 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ed83a428b4fa9bb39c8d1cdfefc08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } } }
I20260928 05:15:54.580210 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:54.583166 21187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:54.583554 21201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:54.589659 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:54.568608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42369 (local address 127.0.0.1:46137)
0928 05:15:54.568798 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:54.568816 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:54.569158 (+ 342us) server_negotiation.cc:407] Connection header received
0928 05:15:54.569307 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:54.569313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:54.569393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:54.569512 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:54.571376 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:54.572847 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:54.575058 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:54.575916 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:54.576116 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:54.581103 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:54.581114 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:54.581128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:15:54.581194 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:54.585048 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:54.586207 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:54.586217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:54.586221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:54.586414 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:15:54.586947 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:15:54.586957 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:15:54.586960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:15:54.587448 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:54.587793 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:54.589037 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0928 05:15:54.589288 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 05:15:54.590179 21201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:54.602578 21201 catalog_manager.cc:1347] Generated new cluster ID: fcbe3b1341f04c128041a8943238f61e
I20260928 05:15:54.602725 21201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:54.628363 21201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:54.630457 21201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:54.647866 21201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8ed83a428b4fa9bb39c8d1cdfefc08: Generated new TSK 0
I20260928 05:15:54.648805 21201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:54.663719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21120
I20260928 05:15:54.664889 21182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:15:54.760411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21120
2026-09-28T05:15:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:54 dist-test-slave-6718 krb5kdc[21219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:54 dist-test-slave-6718 krb5kdc[21219](info): set up 2 sockets
Sep 28 05:15:54 dist-test-slave-6718 krb5kdc[21219](info): commencing operation
krb5kdc: starting...
W20260928 05:15:55.854941 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:56 dist-test-slave-6718 krb5kdc[21219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:56.105652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:56.439229 21232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:56.439591 21232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:56.439754 21232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:56.447257 21232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:56.447413 21232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:56.447522 21232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:56.447635 21232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:56.447714 21232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:56.457674 21232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:56.459038 21232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:56.460341 21232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:56.472146 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:56.472177 21240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:56.472121 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:56.472930 21232 server_base.cc:1034] running on GCE node
I20260928 05:15:56.473701 21232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:56.475088 21232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:56.476318 21232 hybrid_clock.cc:648] HybridClock initialized: now 1790572556476255 us; error 53 us; skew 500 ppm
I20260928 05:15:56.476665 21232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:56 dist-test-slave-6718 krb5kdc[21219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:56.479894 21232 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:15:56.481464 21232 webserver.cc:460] Webserver started at http://127.0.0.1:46255/ using document root <none> and password file <none>
I20260928 05:15:56.482155 21232 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:56.482285 21232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:56.482668 21232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:56.485617 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "ac81f61a3d5e4281ae936fc8a0e669cf"
format_stamp: "Formatted at 2026-09-28 05:15:56 on dist-test-slave-6718"
server_key: "5dca9e06a56c0f300693da28233a8e64"
server_key_iv: "aede08316d7123b453a73edb50847ea8"
server_key_version: "encryptionkey@0"
I20260928 05:15:56.486387 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac81f61a3d5e4281ae936fc8a0e669cf"
format_stamp: "Formatted at 2026-09-28 05:15:56 on dist-test-slave-6718"
server_key: "5dca9e06a56c0f300693da28233a8e64"
server_key_iv: "aede08316d7123b453a73edb50847ea8"
server_key_version: "encryptionkey@0"
I20260928 05:15:56.492892 21232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 05:15:56.497519 21247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:56.498947 21232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:15:56.499125 21232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "ac81f61a3d5e4281ae936fc8a0e669cf"
format_stamp: "Formatted at 2026-09-28 05:15:56 on dist-test-slave-6718"
server_key: "5dca9e06a56c0f300693da28233a8e64"
server_key_iv: "aede08316d7123b453a73edb50847ea8"
server_key_version: "encryptionkey@0"
I20260928 05:15:56.499334 21232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:56.516357 21232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:56.520704 21232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:56.520984 21232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:56.539543 21232 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34413
I20260928 05:15:56.539574 21298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34413 every 8 connection(s)
I20260928 05:15:56.540970 21232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:56.544153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21232
I20260928 05:15:56.544366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:56.544912 388 external_mini_cluster.cc:1428] Setting key 77e0b42c8f46251a2cb9f0020910a44e
I20260928 05:15:56.545776 21299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:56.559439 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:56.546895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35907 (local address 127.0.0.1:34413)
0928 05:15:56.547770 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:56.547783 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:56.547822 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:15:56.549457 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:56.549480 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:56.549933 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:56.550306 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:56.550996 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:56.552416 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:56.554310 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:56.555095 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:56.555237 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:56.556825 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:56.556833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:56.556849 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:56.557473 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:56.557554 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:56.557581 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:56.557688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:56.557948 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 05:15:56.558297 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":427,"threads_started":1}
I20260928 05:15:56.560750 21299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf: Bootstrap starting.
I20260928 05:15:56.563448 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:15:56.564157 21299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:56.565369 21299 log.cc:824] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:56.568421 21299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf: No bootstrap required, opened a new log
I20260928 05:15:56.576005 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:56.568639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49381 (local address 127.0.0.1:34413)
0928 05:15:56.568864 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:56.568873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:56.569320 (+ 447us) server_negotiation.cc:407] Connection header received
0928 05:15:56.569532 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:56.569535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:56.569636 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:56.569717 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:56.570354 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:56.571198 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:56.573299 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:15:56.573757 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:15:56.573857 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:15:56.575238 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:56.575243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:56.575246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:56.575267 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:56.575304 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:56.575318 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:56.575403 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:56.575564 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 05:15:56.575652 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 05:15:56.579599 21299 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 05:15:56.580031 21299 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:56.580173 21299 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac81f61a3d5e4281ae936fc8a0e669cf, State: Initialized, Role: FOLLOWER
I20260928 05:15:56.580925 21299 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 05:15:56.581215 21299 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:56.581383 21299 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:56.581591 21299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:56.584349 21299 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 05:15:56.584901 21299 leader_election.cc:302] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac81f61a3d5e4281ae936fc8a0e669cf; no voters:
I20260928 05:15:56.585489 21299 leader_election.cc:288] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:56.585595 21309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:56.586385 21309 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [term 1 LEADER]: Becoming Leader. State: Replica: ac81f61a3d5e4281ae936fc8a0e669cf, State: Running, Role: LEADER
I20260928 05:15:56.586978 21309 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 05:15:56.587848 21299 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:56.589823 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac81f61a3d5e4281ae936fc8a0e669cf. Latest consensus state: current_term: 1 leader_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } } }
I20260928 05:15:56.590310 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:56.590808 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f61a3d5e4281ae936fc8a0e669cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } } }
I20260928 05:15:56.591243 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:56.595026 21314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:56.600399 21314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:56.609151 21314 catalog_manager.cc:1347] Generated new cluster ID: 97c0701a343b47459b2b96e97363f917
I20260928 05:15:56.609280 21314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:56.628628 21314 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:56.630792 21314 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:56.651670 21314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac81f61a3d5e4281ae936fc8a0e669cf: Generated new TSK 0
I20260928 05:15:56.652827 21314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:56.664799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21232
W20260928 05:15:56.666040 21294 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35907
W20260928 05:15:56.666893 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21232: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:15:56.667054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21232
2026-09-28T05:15:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:56 dist-test-slave-6718 krb5kdc[21330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:56 dist-test-slave-6718 krb5kdc[21330](info): set up 2 sockets
Sep 28 05:15:56 dist-test-slave-6718 krb5kdc[21330](info): commencing operation
krb5kdc: starting...
W20260928 05:15:57.790933 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:57 dist-test-slave-6718 krb5kdc[21330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:58.050490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:15:58.388120 21343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:15:58.388465 21343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:15:58.388650 21343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:15:58.396119 21343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:15:58.396288 21343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:15:58.396401 21343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:15:58.396495 21343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:15:58.396572 21343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:15:58.406977 21343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:15:58.408366 21343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:15:58.409665 21343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:15:58.421983 21348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:58.421954 21351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:15:58.421919 21349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:15:58.423982 21343 server_base.cc:1034] running on GCE node
I20260928 05:15:58.424855 21343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:15:58.426669 21343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:15:58.427892 21343 hybrid_clock.cc:648] HybridClock initialized: now 1790572558427856 us; error 29 us; skew 500 ppm
I20260928 05:15:58.428284 21343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:15:58 dist-test-slave-6718 krb5kdc[21330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:15:58.436570 21343 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:15:58.438202 21343 webserver.cc:460] Webserver started at http://127.0.0.1:42103/ using document root <none> and password file <none>
I20260928 05:15:58.438977 21343 fs_manager.cc:359] Metadata directory not provided
I20260928 05:15:58.439123 21343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:15:58.439433 21343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:15:58.442425 21343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "0d4acddaf631487881e51d69e2130e19"
format_stamp: "Formatted at 2026-09-28 05:15:58 on dist-test-slave-6718"
server_key: "a0ce42e8707f2d44625843cd8664e285"
server_key_iv: "04adf7e5342ef638959f2bd99aa5562e"
server_key_version: "encryptionkey@0"
I20260928 05:15:58.443286 21343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d4acddaf631487881e51d69e2130e19"
format_stamp: "Formatted at 2026-09-28 05:15:58 on dist-test-slave-6718"
server_key: "a0ce42e8707f2d44625843cd8664e285"
server_key_iv: "04adf7e5342ef638959f2bd99aa5562e"
server_key_version: "encryptionkey@0"
I20260928 05:15:58.448468 21343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:15:58.452008 21358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:15:58.453334 21343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:15:58.453557 21343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "0d4acddaf631487881e51d69e2130e19"
format_stamp: "Formatted at 2026-09-28 05:15:58 on dist-test-slave-6718"
server_key: "a0ce42e8707f2d44625843cd8664e285"
server_key_iv: "04adf7e5342ef638959f2bd99aa5562e"
server_key_version: "encryptionkey@0"
I20260928 05:15:58.453773 21343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:15:58.459789 21343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:15:58.464064 21343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:15:58.464357 21343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:15:58.483014 21343 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40271
I20260928 05:15:58.483059 21409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40271 every 8 connection(s)
I20260928 05:15:58.484437 21343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:15:58.484987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21343
I20260928 05:15:58.485172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:15:58.485781 388 external_mini_cluster.cc:1428] Setting key 8ae468c25a55076e487269e7ac4ec8af
I20260928 05:15:58.488869 21410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:15:58.493508 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:58.487604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35681 (local address 127.0.0.1:40271)
0928 05:15:58.488342 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:58.488354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:58.488393 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:15:58.490082 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:58.490105 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:58.490459 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:58.490922 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:58.491971 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:58.491977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:58.491988 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:58.492001 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:58.492068 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:58.492086 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:58.492322 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:58.492466 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 05:15:58.492637 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":324,"threads_started":1}
I20260928 05:15:58.497272 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:15:58.500568 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19: Bootstrap starting.
I20260928 05:15:58.504272 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:15:58.501550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55971 (local address 127.0.0.1:40271)
0928 05:15:58.501759 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 05:15:58.501768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:15:58.502317 (+ 549us) server_negotiation.cc:407] Connection header received
0928 05:15:58.502471 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:15:58.502474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:15:58.502602 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:15:58.502690 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:15:58.503617 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:15:58.503621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:15:58.503624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:15:58.503639 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:15:58.503676 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:15:58.503691 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:15:58.503780 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 05:15:58.503917 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 05:15:58.503961 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 05:15:58.504549 21410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19: Neither blocks nor log segments found. Creating new log.
I20260928 05:15:58.505704 21410 log.cc:824] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19: Log is configured to *not* fsync() on all Append() calls
I20260928 05:15:58.508430 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19: No bootstrap required, opened a new log
I20260928 05:15:58.516026 21410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4acddaf631487881e51d69e2130e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40271 } }
I20260928 05:15:58.516409 21410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:15:58.516515 21410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4acddaf631487881e51d69e2130e19, State: Initialized, Role: FOLLOWER
I20260928 05:15:58.517133 21410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4acddaf631487881e51d69e2130e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40271 } }
I20260928 05:15:58.517364 21410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:15:58.517503 21410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:15:58.517695 21410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:15:58.520164 21410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4acddaf631487881e51d69e2130e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40271 } }
I20260928 05:15:58.520664 21410 leader_election.cc:302] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d4acddaf631487881e51d69e2130e19; no voters:
I20260928 05:15:58.521214 21410 leader_election.cc:288] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:15:58.521381 21420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:15:58.522115 21420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [term 1 LEADER]: Becoming Leader. State: Replica: 0d4acddaf631487881e51d69e2130e19, State: Running, Role: LEADER
I20260928 05:15:58.522712 21420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4acddaf631487881e51d69e2130e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40271 } }
I20260928 05:15:58.523363 21410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:15:58.525671 21421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d4acddaf631487881e51d69e2130e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4acddaf631487881e51d69e2130e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40271 } } }
I20260928 05:15:58.526191 21421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:58.526608 21422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d4acddaf631487881e51d69e2130e19. Latest consensus state: current_term: 1 leader_uuid: "0d4acddaf631487881e51d69e2130e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4acddaf631487881e51d69e2130e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40271 } } }
I20260928 05:15:58.526971 21422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19 [sys.catalog]: This master's current role is: LEADER
I20260928 05:15:58.528110 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:15:58.532473 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:15:58.541044 21429 catalog_manager.cc:1347] Generated new cluster ID: 6ecb5c61c6d24127b84b5f9a0100554c
I20260928 05:15:58.541199 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:15:58.555408 21429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:15:58.555877 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:15:58.564713 21429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d4acddaf631487881e51d69e2130e19: Generated new TSK 0
I20260928 05:15:58.565551 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:15:58.573536 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21343
W20260928 05:15:58.574949 21405 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35681
W20260928 05:15:58.575724 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:15:58.575870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21343
2026-09-28T05:15:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:15:58 dist-test-slave-6718 krb5kdc[21441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:15:58 dist-test-slave-6718 krb5kdc[21441](info): set up 2 sockets
Sep 28 05:15:58 dist-test-slave-6718 krb5kdc[21441](info): commencing operation
krb5kdc: starting...
W20260928 05:15:59.682993 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:15:59 dist-test-slave-6718 krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:15:59.937390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:00.271773 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:00.272107 21454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:00.272264 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:00.279656 21454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:00.279769 21454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:00.279865 21454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:00.279963 21454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:00.280040 21454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:00.289812 21454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:00.291177 21454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:00.292510 21454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:00.304008 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:00.303972 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:00.303953 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:00.305161 21454 server_base.cc:1034] running on GCE node
I20260928 05:16:00.305886 21454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:00.307353 21454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:00.308565 21454 hybrid_clock.cc:648] HybridClock initialized: now 1790572560308508 us; error 42 us; skew 500 ppm
I20260928 05:16:00.308921 21454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:00 dist-test-slave-6718 krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:00.312125 21454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:16:00.313637 21454 webserver.cc:460] Webserver started at http://127.0.0.1:45951/ using document root <none> and password file <none>
I20260928 05:16:00.314378 21454 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:00.314554 21454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:00.314879 21454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:00.317826 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "351da8fd8f334a2990c82cb0f6b42d27"
format_stamp: "Formatted at 2026-09-28 05:16:00 on dist-test-slave-6718"
server_key: "8101d4542158658786d32c7bde39b7b8"
server_key_iv: "6de84fd13472f3c5ef5a4b5165e55111"
server_key_version: "encryptionkey@0"
I20260928 05:16:00.318650 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "351da8fd8f334a2990c82cb0f6b42d27"
format_stamp: "Formatted at 2026-09-28 05:16:00 on dist-test-slave-6718"
server_key: "8101d4542158658786d32c7bde39b7b8"
server_key_iv: "6de84fd13472f3c5ef5a4b5165e55111"
server_key_version: "encryptionkey@0"
I20260928 05:16:00.323727 21454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 05:16:00.327236 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:00.328660 21454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:16:00.328859 21454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "351da8fd8f334a2990c82cb0f6b42d27"
format_stamp: "Formatted at 2026-09-28 05:16:00 on dist-test-slave-6718"
server_key: "8101d4542158658786d32c7bde39b7b8"
server_key_iv: "6de84fd13472f3c5ef5a4b5165e55111"
server_key_version: "encryptionkey@0"
I20260928 05:16:00.329061 21454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:00.353955 21454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:00.358500 21454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:00.358846 21454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:00.377547 21454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40851
I20260928 05:16:00.377581 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40851 every 8 connection(s)
I20260928 05:16:00.378995 21454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:00.382056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21454
I20260928 05:16:00.382267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:00.382834 388 external_mini_cluster.cc:1428] Setting key ab2bfe7e0b724fadacf90651f4139d92
I20260928 05:16:00.383162 21521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:00.398000 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27: Bootstrap starting.
Sep 28 05:16:00 dist-test-slave-6718 krb5kdc[21441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:16:00.401871 21521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:00.403280 21521 log.cc:824] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:00.406673 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27: No bootstrap required, opened a new log
I20260928 05:16:00.409035 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:00.384694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:35731 (local address 127.0.0.1:40851)
0928 05:16:00.385470 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:00.385486 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:00.385536 (+ 50us) server_negotiation.cc:407] Connection header received
0928 05:16:00.388566 (+ 3030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:00.388598 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:00.389191 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:00.389661 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:00.391290 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:00.393208 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:00.394514 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:00.395665 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:00.395858 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:00.400207 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:00.400218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:00.400237 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:00.400288 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:00.404339 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:00.405235 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:00.405243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:00.405251 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:00.405449 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:00.405888 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:00.405892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:00.405895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:00.406580 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:00.406901 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:00.407366 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 05:16:00.407702 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":319,"threads_started":1}
I20260928 05:16:00.412937 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:00.415444 21521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351da8fd8f334a2990c82cb0f6b42d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260928 05:16:00.415871 21521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:00.416046 21521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351da8fd8f334a2990c82cb0f6b42d27, State: Initialized, Role: FOLLOWER
I20260928 05:16:00.416652 21521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351da8fd8f334a2990c82cb0f6b42d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260928 05:16:00.417017 21521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:00.417196 21521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:00.417403 21521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:00.419838 21521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351da8fd8f334a2990c82cb0f6b42d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260928 05:16:00.420336 21521 leader_election.cc:302] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 351da8fd8f334a2990c82cb0f6b42d27; no voters:
I20260928 05:16:00.421000 21521 leader_election.cc:288] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:00.421098 21531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:00.423657 21521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:00.428110 21531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [term 1 LEADER]: Becoming Leader. State: Replica: 351da8fd8f334a2990c82cb0f6b42d27, State: Running, Role: LEADER
I20260928 05:16:00.428932 21531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351da8fd8f334a2990c82cb0f6b42d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260928 05:16:00.431048 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:00.417304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:46679 (local address 127.0.0.1:40851)
0928 05:16:00.417496 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:00.417504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:00.417848 (+ 344us) server_negotiation.cc:407] Connection header received
0928 05:16:00.418034 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:00.418040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:00.418123 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:00.418223 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:00.419666 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:00.420522 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:00.422129 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:00.422632 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:00.422760 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:00.425235 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:00.425240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:00.425244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:00.425273 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:00.428222 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:00.428994 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:00.428999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:00.429003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:00.429133 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:00.429546 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:00.429551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:00.429554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:00.429915 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:00.430193 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:00.430576 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 05:16:00.430763 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 05:16:00.436726 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "351da8fd8f334a2990c82cb0f6b42d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351da8fd8f334a2990c82cb0f6b42d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } } }
I20260928 05:16:00.436693 21533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 351da8fd8f334a2990c82cb0f6b42d27. Latest consensus state: current_term: 1 leader_uuid: "351da8fd8f334a2990c82cb0f6b42d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351da8fd8f334a2990c82cb0f6b42d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } } }
I20260928 05:16:00.437187 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:00.438256 21533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:00.439205 21539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:00.444195 21539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:00.452077 21539 catalog_manager.cc:1347] Generated new cluster ID: bce3e268f90e49979a89fa63673229d6
I20260928 05:16:00.452194 21539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:00.461118 21539 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:00.462966 21539 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:00.473917 21539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 351da8fd8f334a2990c82cb0f6b42d27: Generated new TSK 0
I20260928 05:16:00.475054 21539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:00.481367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21454
I20260928 05:16:00.483035 21516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:00.572409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21454
W20260928 05:16:00.589414 21439 connection.cc:535] client connection to 127.0.0.1:40851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40851): Connection reset by peer (error 104)
2026-09-28T05:16:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:00 dist-test-slave-6718 krb5kdc[21553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:00 dist-test-slave-6718 krb5kdc[21553](info): set up 2 sockets
Sep 28 05:16:00 dist-test-slave-6718 krb5kdc[21553](info): commencing operation
krb5kdc: starting...
W20260928 05:16:01.666937 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:01 dist-test-slave-6718 krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:16:01.919792 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:02.246266 21566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:02.246557 21566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:02.246728 21566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:02.254021 21566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:02.254142 21566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:02.254240 21566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:02.254356 21566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:02.254431 21566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:02.264559 21566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:02.265974 21566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:02.267580 21566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:02.279811 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:02.279968 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:02.279943 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:02.281173 21566 server_base.cc:1034] running on GCE node
I20260928 05:16:02.282032 21566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:02.283521 21566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:02.284808 21566 hybrid_clock.cc:648] HybridClock initialized: now 1790572562284745 us; error 37 us; skew 500 ppm
I20260928 05:16:02.285166 21566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:02 dist-test-slave-6718 krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:02.288396 21566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:16:02.289848 21566 webserver.cc:460] Webserver started at http://127.0.0.1:34779/ using document root <none> and password file <none>
I20260928 05:16:02.290629 21566 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:02.290769 21566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:02.291069 21566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:02.294059 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "a6194c870f8843cba4aa58c330ad26a8"
format_stamp: "Formatted at 2026-09-28 05:16:02 on dist-test-slave-6718"
server_key: "b3ceb16cd5537091804057a8bd4ce1ae"
server_key_iv: "fca8c0077ac512102a0b736e9c423519"
server_key_version: "encryptionkey@0"
I20260928 05:16:02.294875 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "a6194c870f8843cba4aa58c330ad26a8"
format_stamp: "Formatted at 2026-09-28 05:16:02 on dist-test-slave-6718"
server_key: "b3ceb16cd5537091804057a8bd4ce1ae"
server_key_iv: "fca8c0077ac512102a0b736e9c423519"
server_key_version: "encryptionkey@0"
I20260928 05:16:02.299862 21566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:16:02.303332 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:02.304570 21566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 05:16:02.304746 21566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "a6194c870f8843cba4aa58c330ad26a8"
format_stamp: "Formatted at 2026-09-28 05:16:02 on dist-test-slave-6718"
server_key: "b3ceb16cd5537091804057a8bd4ce1ae"
server_key_iv: "fca8c0077ac512102a0b736e9c423519"
server_key_version: "encryptionkey@0"
I20260928 05:16:02.304934 21566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:02.319167 21566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:02.323915 21566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:02.324190 21566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:02.342752 21566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40039
I20260928 05:16:02.342772 21632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40039 every 8 connection(s)
I20260928 05:16:02.344175 21566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:02.347851 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:02.353883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21566
I20260928 05:16:02.354087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:02.354521 388 external_mini_cluster.cc:1428] Setting key 99e49b46ff795abbaa6a7d829766cb84
I20260928 05:16:02.358549 21633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8: Bootstrap starting.
I20260928 05:16:02.362006 21633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:02.363267 21633 log.cc:824] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:02.366492 21633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8: No bootstrap required, opened a new log
I20260928 05:16:02.368229 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:02.356253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:40863 (local address 127.0.0.1:40039)
0928 05:16:02.357025 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:02.357036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:02.357076 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:16:02.358848 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:02.358870 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:02.359256 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:02.359574 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:02.360220 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:02.361535 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:02.363495 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:02.364592 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:02.364736 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:02.365957 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:02.365964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:02.365978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:02.366482 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:02.366627 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:02.366664 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:02.366777 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:02.367074 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 05:16:02.367268 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":371,"threads_started":1}
I20260928 05:16:02.371958 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:02.375696 21633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6194c870f8843cba4aa58c330ad26a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260928 05:16:02.376205 21633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:02.376312 21633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6194c870f8843cba4aa58c330ad26a8, State: Initialized, Role: FOLLOWER
I20260928 05:16:02.376963 21633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6194c870f8843cba4aa58c330ad26a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260928 05:16:02.377252 21633 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:02.377471 21633 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:02.377679 21633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:02.380904 21633 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6194c870f8843cba4aa58c330ad26a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260928 05:16:02.381520 21633 leader_election.cc:302] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6194c870f8843cba4aa58c330ad26a8; no voters:
I20260928 05:16:02.382315 21633 leader_election.cc:288] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:02.382450 21643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:02.384649 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:02.377217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:55451 (local address 127.0.0.1:40039)
0928 05:16:02.377434 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:02.377440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:02.377799 (+ 359us) server_negotiation.cc:407] Connection header received
0928 05:16:02.377999 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:02.378002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:02.378078 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:02.378155 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:02.378785 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:02.379767 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:02.381641 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:02.382191 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:02.382357 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:02.383649 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:02.383654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:02.383658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:02.383678 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:02.383763 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:02.383785 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:02.383882 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:02.384340 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 05:16:02.384397 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:16:02.385478 21633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:02.386786 21643 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [term 1 LEADER]: Becoming Leader. State: Replica: a6194c870f8843cba4aa58c330ad26a8, State: Running, Role: LEADER
I20260928 05:16:02.387507 21643 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6194c870f8843cba4aa58c330ad26a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } }
I20260928 05:16:02.390722 21645 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6194c870f8843cba4aa58c330ad26a8. Latest consensus state: current_term: 1 leader_uuid: "a6194c870f8843cba4aa58c330ad26a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6194c870f8843cba4aa58c330ad26a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } } }
I20260928 05:16:02.391142 21645 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:02.394240 21650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:02.394089 21644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6194c870f8843cba4aa58c330ad26a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6194c870f8843cba4aa58c330ad26a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40039 } } }
I20260928 05:16:02.394518 21644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:02.401603 21650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:02.412027 21650 catalog_manager.cc:1347] Generated new cluster ID: 50c21e31068a4ca492a4bf0a2ceb63ac
I20260928 05:16:02.412217 21650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:02.427609 21650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:02.429286 21650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:02.437973 21650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6194c870f8843cba4aa58c330ad26a8: Generated new TSK 0
I20260928 05:16:02.438815 21650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:02.456132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21566
W20260928 05:16:02.457338 21628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:40863
W20260928 05:16:02.458070 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:02.458227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21566
W20260928 05:16:02.474891 21551 connection.cc:535] client connection to 127.0.0.1:40039 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40039): Connection reset by peer (error 104)
2026-09-28T05:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:02 dist-test-slave-6718 krb5kdc[21664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:02 dist-test-slave-6718 krb5kdc[21664](info): set up 2 sockets
Sep 28 05:16:02 dist-test-slave-6718 krb5kdc[21664](info): commencing operation
krb5kdc: starting...
W20260928 05:16:03.555018 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:03 dist-test-slave-6718 krb5kdc[21664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:16:03.813805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:04.147897 21677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:04.148203 21677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:04.148299 21677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:04.155563 21677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:04.155697 21677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:04.155745 21677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:04.155807 21677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:04.155851 21677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:04.165803 21677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:04.167112 21677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:04.168350 21677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:04.180423 21682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:04.180382 21685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:04.180369 21683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:04.181339 21677 server_base.cc:1034] running on GCE node
I20260928 05:16:04.182193 21677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:04.183753 21677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:04.184976 21677 hybrid_clock.cc:648] HybridClock initialized: now 1790572564184938 us; error 46 us; skew 500 ppm
I20260928 05:16:04.185324 21677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:04 dist-test-slave-6718 krb5kdc[21664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:04.192035 21677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:16:04.193585 21677 webserver.cc:460] Webserver started at http://127.0.0.1:43145/ using document root <none> and password file <none>
I20260928 05:16:04.194276 21677 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:04.194413 21677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:04.194792 21677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:04.197718 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "2a94263c79bb4f92bd61069e6114a337"
format_stamp: "Formatted at 2026-09-28 05:16:04 on dist-test-slave-6718"
server_key: "0785b19d742181630db781d4f9e06509"
server_key_iv: "b1b43810e1044355a0389c2d5b335aa2"
server_key_version: "encryptionkey@0"
I20260928 05:16:04.198459 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a94263c79bb4f92bd61069e6114a337"
format_stamp: "Formatted at 2026-09-28 05:16:04 on dist-test-slave-6718"
server_key: "0785b19d742181630db781d4f9e06509"
server_key_iv: "b1b43810e1044355a0389c2d5b335aa2"
server_key_version: "encryptionkey@0"
I20260928 05:16:04.203533 21677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 05:16:04.206969 21692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:04.208304 21677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 05:16:04.208498 21677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "2a94263c79bb4f92bd61069e6114a337"
format_stamp: "Formatted at 2026-09-28 05:16:04 on dist-test-slave-6718"
server_key: "0785b19d742181630db781d4f9e06509"
server_key_iv: "b1b43810e1044355a0389c2d5b335aa2"
server_key_version: "encryptionkey@0"
I20260928 05:16:04.208703 21677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:04.214390 21677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:04.218204 21677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:04.218487 21677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:04.236764 21677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42817
I20260928 05:16:04.236797 21743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42817 every 8 connection(s)
I20260928 05:16:04.238186 21677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:04.242218 21744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:04.247972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21677
I20260928 05:16:04.248155 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:04.248575 388 external_mini_cluster.cc:1428] Setting key 2daf9bb75e0bab49279dabfed3ca4f23
I20260928 05:16:04.254243 21744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337: Bootstrap starting.
I20260928 05:16:04.257768 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:04.250413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:55889 (local address 127.0.0.1:42817)
0928 05:16:04.251287 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:04.251304 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:04.251362 (+ 58us) server_negotiation.cc:407] Connection header received
0928 05:16:04.253786 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:04.253816 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:04.254282 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:04.254779 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:04.255757 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:04.255766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:04.255784 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:04.255802 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:04.255878 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:04.255909 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:04.256165 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:04.256296 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 05:16:04.256513 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":312,"threads_started":1}
I20260928 05:16:04.258504 21744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:04.259871 21744 log.cc:824] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:04.262157 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:04.263043 21744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337: No bootstrap required, opened a new log
I20260928 05:16:04.269085 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:04.266486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:56803 (local address 127.0.0.1:42817)
0928 05:16:04.266727 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:04.266736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:04.267110 (+ 374us) server_negotiation.cc:407] Connection header received
0928 05:16:04.267312 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:04.267317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:04.267368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:04.267481 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:04.268395 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:04.268404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:04.268408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:04.268423 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:04.268465 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:04.268480 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:04.268612 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:04.268741 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 05:16:04.268812 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 05:16:04.271831 21744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94263c79bb4f92bd61069e6114a337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42817 } }
I20260928 05:16:04.272238 21744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:04.272351 21744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a94263c79bb4f92bd61069e6114a337, State: Initialized, Role: FOLLOWER
I20260928 05:16:04.272914 21744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94263c79bb4f92bd61069e6114a337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42817 } }
I20260928 05:16:04.273195 21744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:04.273356 21744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:04.273532 21744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:04.275900 21744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94263c79bb4f92bd61069e6114a337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42817 } }
I20260928 05:16:04.276485 21744 leader_election.cc:302] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a94263c79bb4f92bd61069e6114a337; no voters:
I20260928 05:16:04.277171 21744 leader_election.cc:288] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:04.277276 21754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:04.277915 21754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [term 1 LEADER]: Becoming Leader. State: Replica: 2a94263c79bb4f92bd61069e6114a337, State: Running, Role: LEADER
I20260928 05:16:04.278450 21754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94263c79bb4f92bd61069e6114a337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42817 } }
I20260928 05:16:04.279521 21744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:04.281414 21755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a94263c79bb4f92bd61069e6114a337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94263c79bb4f92bd61069e6114a337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42817 } } }
I20260928 05:16:04.281816 21755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:04.282272 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a94263c79bb4f92bd61069e6114a337. Latest consensus state: current_term: 1 leader_uuid: "2a94263c79bb4f92bd61069e6114a337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94263c79bb4f92bd61069e6114a337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42817 } } }
I20260928 05:16:04.282584 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:04.284349 21759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:04.289280 21759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:04.298975 21759 catalog_manager.cc:1347] Generated new cluster ID: 15c6bce26a59493eb1ff4fd6c1dd447e
I20260928 05:16:04.299104 21759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:04.324221 21759 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:04.324704 21759 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:04.331936 21759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a94263c79bb4f92bd61069e6114a337: Generated new TSK 0
I20260928 05:16:04.332988 21759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:04.338295 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21677
W20260928 05:16:04.339552 21739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:55889
W20260928 05:16:04.340325 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:04.340471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21677
2026-09-28T05:16:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:04 dist-test-slave-6718 krb5kdc[21775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:04 dist-test-slave-6718 krb5kdc[21775](info): set up 2 sockets
Sep 28 05:16:04 dist-test-slave-6718 krb5kdc[21775](info): commencing operation
krb5kdc: starting...
W20260928 05:16:05.454958 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:05 dist-test-slave-6718 krb5kdc[21775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:16:05.720534 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:06.061998 21788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:06.062391 21788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:06.062587 21788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:06.070122 21788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:06.070297 21788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:06.070403 21788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:06.070504 21788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:06.070613 21788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:06.080973 21788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:06.082368 21788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:06.083724 21788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:06.095715 21793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:06.095678 21796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:06.095748 21794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:06.096536 21788 server_base.cc:1034] running on GCE node
I20260928 05:16:06.097360 21788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:06.098873 21788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:06.100149 21788 hybrid_clock.cc:648] HybridClock initialized: now 1790572566100101 us; error 37 us; skew 500 ppm
I20260928 05:16:06.100571 21788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:06 dist-test-slave-6718 krb5kdc[21775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:06.103766 21788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:16:06.105265 21788 webserver.cc:460] Webserver started at http://127.0.0.1:45479/ using document root <none> and password file <none>
I20260928 05:16:06.106132 21788 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:06.106297 21788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:06.106680 21788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:06.109709 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "0276a75a0f574e638e20f884b7faadef"
format_stamp: "Formatted at 2026-09-28 05:16:06 on dist-test-slave-6718"
server_key: "e2429a4b80f5e504dda4b57d0ef15122"
server_key_iv: "ce263c680ab3926a2d2f0482814344b3"
server_key_version: "encryptionkey@0"
I20260928 05:16:06.110500 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "0276a75a0f574e638e20f884b7faadef"
format_stamp: "Formatted at 2026-09-28 05:16:06 on dist-test-slave-6718"
server_key: "e2429a4b80f5e504dda4b57d0ef15122"
server_key_iv: "ce263c680ab3926a2d2f0482814344b3"
server_key_version: "encryptionkey@0"
I20260928 05:16:06.117501 21788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260928 05:16:06.121146 21803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:06.122624 21788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:16:06.122813 21788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "0276a75a0f574e638e20f884b7faadef"
format_stamp: "Formatted at 2026-09-28 05:16:06 on dist-test-slave-6718"
server_key: "e2429a4b80f5e504dda4b57d0ef15122"
server_key_iv: "ce263c680ab3926a2d2f0482814344b3"
server_key_version: "encryptionkey@0"
I20260928 05:16:06.123014 21788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:06.140286 21788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:06.144604 21788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:06.144908 21788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:06.164289 21788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36115
I20260928 05:16:06.164319 21854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36115 every 8 connection(s)
I20260928 05:16:06.165771 21788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:06.167337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21788
I20260928 05:16:06.167527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:06.168128 388 external_mini_cluster.cc:1428] Setting key c868b061aadfcf2ef78e9f5724db7b08
I20260928 05:16:06.170971 21855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 05:16:06 dist-test-slave-6718 krb5kdc[21775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 05:16:06.185882 21855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef: Bootstrap starting.
I20260928 05:16:06.190917 21855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:06.192384 21855 log.cc:824] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:06.194484 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:06.170046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:36819 (local address 127.0.0.1:36115)
0928 05:16:06.170892 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:06.170903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:06.170943 (+ 40us) server_negotiation.cc:407] Connection header received
0928 05:16:06.172645 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:06.172668 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:06.173073 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:06.173428 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:06.175433 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:06.176917 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:06.179015 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:06.179718 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:06.179860 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:06.185162 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:06.185171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:06.185191 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:06.185238 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:06.189646 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:06.190595 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:06.190609 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:06.190621 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:06.190856 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:06.191373 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:06.191380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:06.191385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:06.192127 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:06.192461 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:06.193063 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 05:16:06.193467 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":358,"threads_started":1}
I20260928 05:16:06.195566 21855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef: No bootstrap required, opened a new log
I20260928 05:16:06.198745 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:06.206202 21855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276a75a0f574e638e20f884b7faadef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 05:16:06.206673 21855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:06.206820 21855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0276a75a0f574e638e20f884b7faadef, State: Initialized, Role: FOLLOWER
I20260928 05:16:06.207438 21855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276a75a0f574e638e20f884b7faadef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 05:16:06.207706 21855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:06.207859 21855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:06.208063 21855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:06.211290 21855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276a75a0f574e638e20f884b7faadef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 05:16:06.211971 21855 leader_election.cc:302] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0276a75a0f574e638e20f884b7faadef; no voters:
I20260928 05:16:06.212687 21855 leader_election.cc:288] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:06.212821 21865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:06.214831 21855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:06.214761 21865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [term 1 LEADER]: Becoming Leader. State: Replica: 0276a75a0f574e638e20f884b7faadef, State: Running, Role: LEADER
I20260928 05:16:06.215394 21865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276a75a0f574e638e20f884b7faadef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 05:16:06.219972 21867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0276a75a0f574e638e20f884b7faadef. Latest consensus state: current_term: 1 leader_uuid: "0276a75a0f574e638e20f884b7faadef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276a75a0f574e638e20f884b7faadef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260928 05:16:06.220376 21867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:06.220985 21866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0276a75a0f574e638e20f884b7faadef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276a75a0f574e638e20f884b7faadef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260928 05:16:06.221289 21866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:06.222461 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:06.204233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:60453 (local address 127.0.0.1:36115)
0928 05:16:06.204445 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:06.204451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:06.204839 (+ 388us) server_negotiation.cc:407] Connection header received
0928 05:16:06.204983 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:06.204990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:06.205085 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:06.205193 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:06.206750 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:06.207756 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:06.209650 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:06.210121 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:06.210217 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:06.213411 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:06.213419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:06.213424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:06.213465 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:06.217714 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:06.219038 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:06.219043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:06.219046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:06.219241 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:06.219892 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:06.219899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:06.219904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:06.220420 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:06.220830 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:06.221698 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 05:16:06.221912 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 05:16:06.232028 21874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:06.237957 21874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:06.247516 21874 catalog_manager.cc:1347] Generated new cluster ID: dfe2780ea16746e39af3a8adf6ff4a81
I20260928 05:16:06.247680 21874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:06.266660 21874 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:06.268067 21874 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:06.276888 21874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0276a75a0f574e638e20f884b7faadef: Generated new TSK 0
I20260928 05:16:06.277949 21874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:06.292354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21788
I20260928 05:16:06.294425 21850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:06.388283 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21788
W20260928 05:16:06.405571 21773 connection.cc:535] client connection to 127.0.0.1:36115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36115): Connection reset by peer (error 104)
2026-09-28T05:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:06 dist-test-slave-6718 krb5kdc[21888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:06 dist-test-slave-6718 krb5kdc[21888](info): set up 2 sockets
Sep 28 05:16:06 dist-test-slave-6718 krb5kdc[21888](info): commencing operation
krb5kdc: starting...
W20260928 05:16:07.510983 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:07 dist-test-slave-6718 krb5kdc[21888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:16:07.765673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:08.095875 21901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:08.096175 21901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:08.096268 21901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:08.103674 21901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:08.103786 21901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:08.103832 21901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:08.103892 21901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:08.103935 21901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:08.117249 21901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:08.118489 21901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:08.119843 21901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:08.131467 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:08.131554 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:08.131533 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:08.132287 21901 server_base.cc:1034] running on GCE node
I20260928 05:16:08.132987 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:08.134368 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:08.135571 21901 hybrid_clock.cc:648] HybridClock initialized: now 1790572568135516 us; error 48 us; skew 500 ppm
I20260928 05:16:08.135944 21901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:08 dist-test-slave-6718 krb5kdc[21888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:08.139012 21901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:16:08.140499 21901 webserver.cc:460] Webserver started at http://127.0.0.1:34077/ using document root <none> and password file <none>
I20260928 05:16:08.141150 21901 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:08.141252 21901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:08.141506 21901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:08.144438 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "6987069ada1c474fbb18432f548d982c"
format_stamp: "Formatted at 2026-09-28 05:16:08 on dist-test-slave-6718"
server_key: "4c91593dacc24afa5c30595de8f9c333"
server_key_iv: "b4fd40c5d94881aad0605190597f66c2"
server_key_version: "encryptionkey@0"
I20260928 05:16:08.145143 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "6987069ada1c474fbb18432f548d982c"
format_stamp: "Formatted at 2026-09-28 05:16:08 on dist-test-slave-6718"
server_key: "4c91593dacc24afa5c30595de8f9c333"
server_key_iv: "b4fd40c5d94881aad0605190597f66c2"
server_key_version: "encryptionkey@0"
I20260928 05:16:08.150197 21901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:16:08.153579 21916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:08.154968 21901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:16:08.155100 21901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "6987069ada1c474fbb18432f548d982c"
format_stamp: "Formatted at 2026-09-28 05:16:08 on dist-test-slave-6718"
server_key: "4c91593dacc24afa5c30595de8f9c333"
server_key_iv: "b4fd40c5d94881aad0605190597f66c2"
server_key_version: "encryptionkey@0"
I20260928 05:16:08.155268 21901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:08.180102 21901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:08.184508 21901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:08.184759 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:08.203593 21901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46615
I20260928 05:16:08.203708 21967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46615 every 8 connection(s)
I20260928 05:16:08.205072 21901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:08.208837 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:08.210261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 21901
I20260928 05:16:08.210453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:08.211081 388 external_mini_cluster.cc:1428] Setting key 66bb731786e860d0761a7377c2d3e919
I20260928 05:16:08.223565 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c: Bootstrap starting.
I20260928 05:16:08.224800 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:08.213020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:60947 (local address 127.0.0.1:46615)
0928 05:16:08.213842 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:08.213854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:08.213898 (+ 44us) server_negotiation.cc:407] Connection header received
0928 05:16:08.215579 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:08.215608 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:08.216051 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:08.216410 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:08.217078 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:08.218370 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:08.220389 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:08.221116 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:08.221262 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:08.222590 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:08.222605 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:08.222620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:08.223167 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:08.223245 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:08.223269 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:08.223383 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:08.223684 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 05:16:08.223849 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":374,"threads_started":1}
I20260928 05:16:08.226898 21968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:08.228081 21968 log.cc:824] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:08.228772 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:08.230923 21968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c: No bootstrap required, opened a new log
I20260928 05:16:08.240129 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:08.233362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:56311 (local address 127.0.0.1:46615)
0928 05:16:08.233576 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:08.233583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:08.233898 (+ 315us) server_negotiation.cc:407] Connection header received
0928 05:16:08.234098 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:08.234102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:08.234168 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:08.234249 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:08.234872 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:08.235662 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:08.237516 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:08.238010 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:08.238108 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:08.239344 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:08.239349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:08.239352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:08.239373 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:08.239426 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:08.239440 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:08.239526 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:08.239741 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 05:16:08.239813 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:16:08.240316 21968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6987069ada1c474fbb18432f548d982c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46615 } }
I20260928 05:16:08.240780 21968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:08.240888 21968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6987069ada1c474fbb18432f548d982c, State: Initialized, Role: FOLLOWER
I20260928 05:16:08.241518 21968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6987069ada1c474fbb18432f548d982c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46615 } }
I20260928 05:16:08.241758 21968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:08.241904 21968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:08.242067 21968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:08.244343 21968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6987069ada1c474fbb18432f548d982c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46615 } }
I20260928 05:16:08.244824 21968 leader_election.cc:302] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6987069ada1c474fbb18432f548d982c; no voters:
I20260928 05:16:08.245393 21968 leader_election.cc:288] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:08.245517 21978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:08.246174 21978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [term 1 LEADER]: Becoming Leader. State: Replica: 6987069ada1c474fbb18432f548d982c, State: Running, Role: LEADER
I20260928 05:16:08.246781 21978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6987069ada1c474fbb18432f548d982c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46615 } }
I20260928 05:16:08.247474 21968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:08.249377 21979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6987069ada1c474fbb18432f548d982c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6987069ada1c474fbb18432f548d982c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46615 } } }
I20260928 05:16:08.249732 21979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:08.251360 21983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:08.251333 21980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6987069ada1c474fbb18432f548d982c. Latest consensus state: current_term: 1 leader_uuid: "6987069ada1c474fbb18432f548d982c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6987069ada1c474fbb18432f548d982c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46615 } } }
I20260928 05:16:08.251634 21980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:08.257148 21983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:08.266448 21983 catalog_manager.cc:1347] Generated new cluster ID: 95418ff57d3b4dca9d2d1320998bb118
I20260928 05:16:08.266579 21983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:08.284008 21983 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:08.285257 21983 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:08.290838 21983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6987069ada1c474fbb18432f548d982c: Generated new TSK 0
I20260928 05:16:08.291642 21983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:08.308473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21901
W20260928 05:16:08.309722 21963 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:60947
W20260928 05:16:08.310485 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 21901: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:08.310680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 21901
W20260928 05:16:08.327499 21886 connection.cc:535] client connection to 127.0.0.1:46615 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46615): Connection reset by peer (error 104)
2026-09-28T05:16:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:08 dist-test-slave-6718 krb5kdc[21999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:08 dist-test-slave-6718 krb5kdc[21999](info): set up 2 sockets
Sep 28 05:16:08 dist-test-slave-6718 krb5kdc[21999](info): commencing operation
krb5kdc: starting...
W20260928 05:16:09.414983 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:09 dist-test-slave-6718 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 05:16:09.661975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:09.994350 22012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:09.994702 22012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:09.994891 22012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:10.002357 22012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:10.002475 22012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:10.002595 22012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:10.002684 22012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:10.002760 22012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:10.012725 22012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44461
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:10.014070 22012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:10.015420 22012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:10.027354 22020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:10.027482 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:10.027513 22018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:10.028862 22012 server_base.cc:1034] running on GCE node
I20260928 05:16:10.029682 22012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:10.031229 22012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:10.032445 22012 hybrid_clock.cc:648] HybridClock initialized: now 1790572570032382 us; error 54 us; skew 500 ppm
I20260928 05:16:10.032812 22012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:10 dist-test-slave-6718 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:10.047199 22012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 05:16:10.048980 22012 webserver.cc:460] Webserver started at http://127.0.0.1:40053/ using document root <none> and password file <none>
I20260928 05:16:10.049734 22012 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:10.049877 22012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:10.050202 22012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:10.053349 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "c0383faba89140e09c5402022414f8ad"
format_stamp: "Formatted at 2026-09-28 05:16:10 on dist-test-slave-6718"
server_key: "4b8ead00d1db01923707db8aec0cd9ba"
server_key_iv: "48678832ee5fee5a949e7810f7cfc524"
server_key_version: "encryptionkey@0"
I20260928 05:16:10.054121 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0383faba89140e09c5402022414f8ad"
format_stamp: "Formatted at 2026-09-28 05:16:10 on dist-test-slave-6718"
server_key: "4b8ead00d1db01923707db8aec0cd9ba"
server_key_iv: "48678832ee5fee5a949e7810f7cfc524"
server_key_version: "encryptionkey@0"
I20260928 05:16:10.059597 22012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 05:16:10.063230 22027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:10.064522 22012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:16:10.064699 22012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "c0383faba89140e09c5402022414f8ad"
format_stamp: "Formatted at 2026-09-28 05:16:10 on dist-test-slave-6718"
server_key: "4b8ead00d1db01923707db8aec0cd9ba"
server_key_iv: "48678832ee5fee5a949e7810f7cfc524"
server_key_version: "encryptionkey@0"
I20260928 05:16:10.064895 22012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:10.070588 22012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:10.074806 22012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:10.075090 22012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:10.092994 22012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44461
I20260928 05:16:10.093036 22078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44461 every 8 connection(s)
I20260928 05:16:10.094400 22012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:10.097564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22012
I20260928 05:16:10.097735 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:10.098259 388 external_mini_cluster.cc:1428] Setting key 61a4872afbf12bb81d2df1a0c626f390
I20260928 05:16:10.098625 22079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:10.106509 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:10.100023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:39005 (local address 127.0.0.1:44461)
0928 05:16:10.100844 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:10.100856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:10.100895 (+ 39us) server_negotiation.cc:407] Connection header received
0928 05:16:10.103112 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:10.103132 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:10.103459 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:10.103792 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:10.104826 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:10.104836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:10.104848 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:10.104865 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:10.104921 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:10.104946 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:10.105115 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:10.105210 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 05:16:10.105409 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":341,"threads_started":1}
I20260928 05:16:10.112047 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:10.114109 22079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad: Bootstrap starting.
I20260928 05:16:10.118975 22079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:10.120359 22079 log.cc:824] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:10.123615 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:10.120569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:42509 (local address 127.0.0.1:44461)
0928 05:16:10.120819 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:10.120828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:10.121621 (+ 793us) server_negotiation.cc:407] Connection header received
0928 05:16:10.121838 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:10.121843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:10.121900 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:10.122018 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:10.122988 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:10.122993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:10.122998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:10.123016 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:10.123067 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:10.123087 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:10.123220 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:10.123265 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 05:16:10.123323 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:16:10.123988 22079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad: No bootstrap required, opened a new log
I20260928 05:16:10.131415 22079 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0383faba89140e09c5402022414f8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260928 05:16:10.131768 22079 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:10.131841 22079 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0383faba89140e09c5402022414f8ad, State: Initialized, Role: FOLLOWER
I20260928 05:16:10.132318 22079 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0383faba89140e09c5402022414f8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260928 05:16:10.132506 22079 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:10.132601 22079 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:10.132725 22079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:10.134819 22079 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0383faba89140e09c5402022414f8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260928 05:16:10.135349 22079 leader_election.cc:302] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0383faba89140e09c5402022414f8ad; no voters:
I20260928 05:16:10.135861 22079 leader_election.cc:288] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:10.136013 22089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:10.136687 22089 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [term 1 LEADER]: Becoming Leader. State: Replica: c0383faba89140e09c5402022414f8ad, State: Running, Role: LEADER
I20260928 05:16:10.137271 22089 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0383faba89140e09c5402022414f8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260928 05:16:10.137853 22079 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:10.140030 22091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0383faba89140e09c5402022414f8ad. Latest consensus state: current_term: 1 leader_uuid: "c0383faba89140e09c5402022414f8ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0383faba89140e09c5402022414f8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } } }
I20260928 05:16:10.140491 22091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:10.141358 22098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:10.141331 22090 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0383faba89140e09c5402022414f8ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0383faba89140e09c5402022414f8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } } }
I20260928 05:16:10.142474 22090 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:10.146477 22098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:10.155189 22098 catalog_manager.cc:1347] Generated new cluster ID: 213d9477cddc43b183fc137a531060a7
I20260928 05:16:10.155303 22098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:10.168908 22098 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:10.169442 22098 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:10.178570 22098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0383faba89140e09c5402022414f8ad: Generated new TSK 0
I20260928 05:16:10.179602 22098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:10.192163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22012
W20260928 05:16:10.193388 22074 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:39005
W20260928 05:16:10.194132 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:10.194298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22012
2026-09-28T05:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:10 dist-test-slave-6718 krb5kdc[22110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:10 dist-test-slave-6718 krb5kdc[22110](info): set up 2 sockets
Sep 28 05:16:10 dist-test-slave-6718 krb5kdc[22110](info): commencing operation
krb5kdc: starting...
W20260928 05:16:11.303089 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:11 dist-test-slave-6718 krb5kdc[22110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:11.562619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:11.903992 22123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:11.904292 22123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:11.904474 22123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:11.911854 22123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:11.911983 22123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:11.912068 22123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:11.912185 22123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:11.912266 22123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:11.922451 22123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:11.923812 22123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:11.925120 22123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:11.936918 22128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:11.936987 22131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:11.936955 22129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:11.938017 22123 server_base.cc:1034] running on GCE node
I20260928 05:16:11.938887 22123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:11.940441 22123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:11.941695 22123 hybrid_clock.cc:648] HybridClock initialized: now 1790572571941616 us; error 90 us; skew 500 ppm
I20260928 05:16:11.942076 22123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:11 dist-test-slave-6718 krb5kdc[22110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:11.945478 22123 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:11.947201 22123 webserver.cc:460] Webserver started at http://127.0.97.62:38643/ using document root <none> and password file <none>
I20260928 05:16:11.947992 22123 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:11.948139 22123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:11.948446 22123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:11.951563 22123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "2badcea761484fc0a94c7a469c14332c"
format_stamp: "Formatted at 2026-09-28 05:16:11 on dist-test-slave-6718"
server_key: "29c95c8fc9c7374eca629edd3784b898"
server_key_iv: "1539232f8b9b74869532e4a7957ac738"
server_key_version: "encryptionkey@0"
I20260928 05:16:11.952364 22123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "2badcea761484fc0a94c7a469c14332c"
format_stamp: "Formatted at 2026-09-28 05:16:11 on dist-test-slave-6718"
server_key: "29c95c8fc9c7374eca629edd3784b898"
server_key_iv: "1539232f8b9b74869532e4a7957ac738"
server_key_version: "encryptionkey@0"
I20260928 05:16:11.957757 22123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:16:11.961347 22138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:11.962714 22123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:16:11.962921 22123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "2badcea761484fc0a94c7a469c14332c"
format_stamp: "Formatted at 2026-09-28 05:16:11 on dist-test-slave-6718"
server_key: "29c95c8fc9c7374eca629edd3784b898"
server_key_iv: "1539232f8b9b74869532e4a7957ac738"
server_key_version: "encryptionkey@0"
I20260928 05:16:11.963126 22123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:11.988390 22123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:11.992955 22123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:11.993250 22123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:12.011772 22123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36621
I20260928 05:16:12.011797 22189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36621 every 8 connection(s)
I20260928 05:16:12.013231 22123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:12.017179 22190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:12.019104 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22123
I20260928 05:16:12.019316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:12.019734 388 external_mini_cluster.cc:1428] Setting key 03e376a5e3ed1d64e048b4f71dae92b2
I20260928 05:16:12.030042 22190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c: Bootstrap starting.
I20260928 05:16:12.034314 22190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c: Neither blocks nor log segments found. Creating new log.
Sep 28 05:16:12 dist-test-slave-6718 krb5kdc[22110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:16:12.035550 22190 log.cc:824] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:12.038992 22190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c: No bootstrap required, opened a new log
I20260928 05:16:12.044307 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:12.021810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60871 (local address 127.0.97.62:36621)
0928 05:16:12.022725 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:12.022737 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:12.022783 (+ 46us) server_negotiation.cc:407] Connection header received
0928 05:16:12.024502 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:12.024527 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:12.024954 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:12.025294 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:12.027220 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:12.028628 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:12.030564 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:12.031308 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:12.031444 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:12.036546 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:12.036557 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:12.036579 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:12.036627 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:12.039910 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:12.040827 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:12.040836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:12.040842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:12.041000 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:12.041475 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:12.041478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:12.041480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:12.042000 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:12.042232 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:12.042861 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 05:16:12.043249 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":430,"threads_started":1}
I20260928 05:16:12.048588 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:12.048401 22190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2badcea761484fc0a94c7a469c14332c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36621 } }
I20260928 05:16:12.049024 22190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:12.049152 22190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2badcea761484fc0a94c7a469c14332c, State: Initialized, Role: FOLLOWER
I20260928 05:16:12.049793 22190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2badcea761484fc0a94c7a469c14332c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36621 } }
I20260928 05:16:12.050032 22190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:12.050166 22190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:12.050345 22190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:12.052624 22190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2badcea761484fc0a94c7a469c14332c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36621 } }
I20260928 05:16:12.053229 22190 leader_election.cc:302] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2badcea761484fc0a94c7a469c14332c; no voters:
I20260928 05:16:12.054016 22190 leader_election.cc:288] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:12.054126 22200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:12.054914 22200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [term 1 LEADER]: Becoming Leader. State: Replica: 2badcea761484fc0a94c7a469c14332c, State: Running, Role: LEADER
I20260928 05:16:12.055538 22200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2badcea761484fc0a94c7a469c14332c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36621 } }
I20260928 05:16:12.058362 22190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:12.058434 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2badcea761484fc0a94c7a469c14332c. Latest consensus state: current_term: 1 leader_uuid: "2badcea761484fc0a94c7a469c14332c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2badcea761484fc0a94c7a469c14332c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36621 } } }
I20260928 05:16:12.058884 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:12.058741 22202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2badcea761484fc0a94c7a469c14332c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2badcea761484fc0a94c7a469c14332c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36621 } } }
I20260928 05:16:12.059026 22202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:12.065136 22204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:12.074818 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:12.053234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34453 (local address 127.0.97.62:36621)
0928 05:16:12.053436 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:12.053444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:12.053889 (+ 445us) server_negotiation.cc:407] Connection header received
0928 05:16:12.054109 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:12.054115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:12.054201 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:12.054315 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:12.056411 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:12.057337 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:12.059295 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:12.060080 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:12.060260 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:12.064694 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:12.064704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:12.064709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:12.064751 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:12.069324 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:12.070673 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:12.070682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:12.070686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:12.070866 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:12.071572 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:12.071580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:12.071586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:12.072067 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:12.072396 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:12.074011 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0928 05:16:12.074259 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 05:16:12.080679 22204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:12.089751 22204 catalog_manager.cc:1347] Generated new cluster ID: e310c7df71ab4a0b8e4be5e0c0db27ee
I20260928 05:16:12.089875 22204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:12.104331 22204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:12.106181 22204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:12.114099 22204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2badcea761484fc0a94c7a469c14332c: Generated new TSK 0
I20260928 05:16:12.115238 22204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:12.129631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22123
I20260928 05:16:12.130820 22185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:12.224854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22123
W20260928 05:16:12.242029 22108 connection.cc:535] client connection to 127.0.97.62:36621 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36621): Connection reset by peer (error 104)
2026-09-28T05:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:12 dist-test-slave-6718 krb5kdc[22222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:12 dist-test-slave-6718 krb5kdc[22222](info): set up 2 sockets
Sep 28 05:16:12 dist-test-slave-6718 krb5kdc[22222](info): commencing operation
krb5kdc: starting...
W20260928 05:16:13.355038 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:13 dist-test-slave-6718 krb5kdc[22222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:13.608145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:13.933584 22235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:13.933949 22235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:13.934125 22235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:13.941530 22235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:13.941709 22235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:13.941838 22235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:13.941951 22235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:13.942030 22235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:13.952170 22235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:13.953517 22235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:13.954869 22235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:13.966560 22240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:13.966511 22243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:13.966471 22241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:13.967312 22235 server_base.cc:1034] running on GCE node
I20260928 05:16:13.968081 22235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:13.969467 22235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:13.970669 22235 hybrid_clock.cc:648] HybridClock initialized: now 1790572573970630 us; error 33 us; skew 500 ppm
I20260928 05:16:13.971045 22235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:13 dist-test-slave-6718 krb5kdc[22222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:13.974310 22235 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:13.975929 22235 webserver.cc:460] Webserver started at http://127.0.97.62:34817/ using document root <none> and password file <none>
I20260928 05:16:13.976619 22235 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:13.976799 22235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:13.977134 22235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:13.980199 22235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "73f7ac46d70a44cf9006c3dd137bb008"
format_stamp: "Formatted at 2026-09-28 05:16:13 on dist-test-slave-6718"
server_key: "11a1879b7fd85a42099fec58d8157831"
server_key_iv: "4392615b87f68e853f4724da49c265ae"
server_key_version: "encryptionkey@0"
I20260928 05:16:13.981001 22235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "73f7ac46d70a44cf9006c3dd137bb008"
format_stamp: "Formatted at 2026-09-28 05:16:13 on dist-test-slave-6718"
server_key: "11a1879b7fd85a42099fec58d8157831"
server_key_iv: "4392615b87f68e853f4724da49c265ae"
server_key_version: "encryptionkey@0"
I20260928 05:16:13.986225 22235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 05:16:13.989825 22250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:13.991237 22235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:16:13.991411 22235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "73f7ac46d70a44cf9006c3dd137bb008"
format_stamp: "Formatted at 2026-09-28 05:16:13 on dist-test-slave-6718"
server_key: "11a1879b7fd85a42099fec58d8157831"
server_key_iv: "4392615b87f68e853f4724da49c265ae"
server_key_version: "encryptionkey@0"
I20260928 05:16:13.991612 22235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:14.007479 22235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:14.012001 22235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:14.012290 22235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:14.030843 22235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40989
I20260928 05:16:14.030876 22301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40989 every 8 connection(s)
I20260928 05:16:14.032295 22235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:14.036147 22302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:14.042044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22235
I20260928 05:16:14.042233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:14.042809 388 external_mini_cluster.cc:1428] Setting key 3b8badb155f2706823b5c672f23f521b
I20260928 05:16:14.047582 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008: Bootstrap starting.
I20260928 05:16:14.050662 22302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:14.051658 22302 log.cc:824] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:14.054875 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008: No bootstrap required, opened a new log
I20260928 05:16:14.055917 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:14.044737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34531 (local address 127.0.97.62:40989)
0928 05:16:14.045564 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:14.045576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:14.045624 (+ 48us) server_negotiation.cc:407] Connection header received
0928 05:16:14.047364 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:14.047387 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:14.047824 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:14.048213 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:14.048737 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:14.050047 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:14.051512 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:14.052375 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:14.052572 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:14.053694 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:14.053702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:14.053720 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:14.054269 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:14.054336 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:14.054359 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:14.054472 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:14.054816 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 05:16:14.054993 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":393,"threads_started":1}
I20260928 05:16:14.059645 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:14.064380 22302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f7ac46d70a44cf9006c3dd137bb008" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260928 05:16:14.064802 22302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:14.064935 22302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f7ac46d70a44cf9006c3dd137bb008, State: Initialized, Role: FOLLOWER
I20260928 05:16:14.065701 22302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f7ac46d70a44cf9006c3dd137bb008" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260928 05:16:14.066016 22302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:14.066180 22302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:14.066361 22302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:14.068750 22302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f7ac46d70a44cf9006c3dd137bb008" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260928 05:16:14.069296 22302 leader_election.cc:302] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f7ac46d70a44cf9006c3dd137bb008; no voters:
I20260928 05:16:14.069965 22302 leader_election.cc:288] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:14.070075 22312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:14.071691 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:14.064938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48825 (local address 127.0.97.62:40989)
0928 05:16:14.065122 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:14.065129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:14.065552 (+ 423us) server_negotiation.cc:407] Connection header received
0928 05:16:14.065730 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:14.065736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:14.065815 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:14.065921 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:14.066595 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:14.067477 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:14.068808 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:14.069323 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:14.069463 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:14.070789 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:14.070795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:14.070799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:14.070818 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:14.070888 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:14.070911 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:14.071021 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:14.071356 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 05:16:14.071411 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:16:14.072599 22302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:14.077044 22312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [term 1 LEADER]: Becoming Leader. State: Replica: 73f7ac46d70a44cf9006c3dd137bb008, State: Running, Role: LEADER
I20260928 05:16:14.077777 22312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f7ac46d70a44cf9006c3dd137bb008" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } }
I20260928 05:16:14.081580 22314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73f7ac46d70a44cf9006c3dd137bb008. Latest consensus state: current_term: 1 leader_uuid: "73f7ac46d70a44cf9006c3dd137bb008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f7ac46d70a44cf9006c3dd137bb008" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } } }
I20260928 05:16:14.081947 22314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:14.081679 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73f7ac46d70a44cf9006c3dd137bb008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f7ac46d70a44cf9006c3dd137bb008" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40989 } } }
I20260928 05:16:14.082410 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:14.082813 22322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:14.088145 22322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:14.097123 22322 catalog_manager.cc:1347] Generated new cluster ID: d4ac3537c448430eaaa038384ce83bb8
I20260928 05:16:14.097244 22322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:14.116318 22322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:14.118209 22322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:14.126863 22322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73f7ac46d70a44cf9006c3dd137bb008: Generated new TSK 0
I20260928 05:16:14.127866 22322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:14.136723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22235
W20260928 05:16:14.137895 22297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34531
W20260928 05:16:14.138643 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22235: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:14.138787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22235
W20260928 05:16:14.155370 22220 connection.cc:535] client connection to 127.0.97.62:40989 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40989): Connection reset by peer (error 104)
2026-09-28T05:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:14 dist-test-slave-6718 krb5kdc[22333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:14 dist-test-slave-6718 krb5kdc[22333](info): set up 2 sockets
Sep 28 05:16:14 dist-test-slave-6718 krb5kdc[22333](info): commencing operation
krb5kdc: starting...
W20260928 05:16:15.239005 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:15 dist-test-slave-6718 krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:15.496011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46201
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:15.820935 22346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:15.821231 22346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:15.821326 22346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:15.828521 22346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:15.828634 22346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:15.828681 22346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:15.828742 22346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:15.828786 22346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:15.838725 22346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:15.840000 22346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:15.841317 22346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:15.853379 22354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:15.853338 22351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:15.853432 22352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:15.859678 22346 server_base.cc:1034] running on GCE node
I20260928 05:16:15.860494 22346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:15.861950 22346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:15.863145 22346 hybrid_clock.cc:648] HybridClock initialized: now 1790572575863101 us; error 37 us; skew 500 ppm
I20260928 05:16:15.863469 22346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:15 dist-test-slave-6718 krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:15.872040 22346 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:15.873770 22346 webserver.cc:460] Webserver started at http://127.0.97.62:35703/ using document root <none> and password file <none>
I20260928 05:16:15.874590 22346 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:15.874701 22346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:15.875032 22346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:15.878127 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "c0139f1947564dc9a95b01496760ebce"
format_stamp: "Formatted at 2026-09-28 05:16:15 on dist-test-slave-6718"
server_key: "52b9f83011c54722c45dee299ae88d3e"
server_key_iv: "ed7888c2ed8a5ca1a89288f01b2f8cfe"
server_key_version: "encryptionkey@0"
I20260928 05:16:15.878965 22346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0139f1947564dc9a95b01496760ebce"
format_stamp: "Formatted at 2026-09-28 05:16:15 on dist-test-slave-6718"
server_key: "52b9f83011c54722c45dee299ae88d3e"
server_key_iv: "ed7888c2ed8a5ca1a89288f01b2f8cfe"
server_key_version: "encryptionkey@0"
I20260928 05:16:15.884377 22346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:16:15.887936 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:15.889349 22346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:16:15.889523 22346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "c0139f1947564dc9a95b01496760ebce"
format_stamp: "Formatted at 2026-09-28 05:16:15 on dist-test-slave-6718"
server_key: "52b9f83011c54722c45dee299ae88d3e"
server_key_iv: "ed7888c2ed8a5ca1a89288f01b2f8cfe"
server_key_version: "encryptionkey@0"
I20260928 05:16:15.889735 22346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:15.895325 22346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:15.899631 22346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:15.899920 22346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:15.918181 22346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46201
I20260928 05:16:15.918222 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46201 every 8 connection(s)
I20260928 05:16:15.919698 22346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:15.920267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22346
I20260928 05:16:15.920555 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:15.921175 388 external_mini_cluster.cc:1428] Setting key 7893d21a3bef6d08ee77c403b0c2a714
I20260928 05:16:15.924551 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:15.930632 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:15.923147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46201 (local address 127.0.97.62:46201)
0928 05:16:15.924238 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:15.924254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:15.924307 (+ 53us) server_negotiation.cc:407] Connection header received
0928 05:16:15.926707 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:15.926736 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:15.927173 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:15.927645 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:15.928652 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:15.928660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:15.928688 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:15.928704 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:15.928790 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:15.928816 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:15.929046 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:15.929191 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 05:16:15.929386 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":526,"threads_started":1}
I20260928 05:16:15.934613 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:15.937888 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce: Bootstrap starting.
I20260928 05:16:15.941759 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:15.938891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35043 (local address 127.0.97.62:46201)
0928 05:16:15.939094 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:15.939103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:15.939651 (+ 548us) server_negotiation.cc:407] Connection header received
0928 05:16:15.939908 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:15.939912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:15.939967 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:15.940054 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:15.941093 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:15.941097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:15.941100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:15.941113 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:15.941148 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:15.941162 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:15.941247 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:15.941384 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 05:16:15.941434 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:16:15.941982 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:15.943316 22413 log.cc:824] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:15.946113 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce: No bootstrap required, opened a new log
I20260928 05:16:15.953807 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0139f1947564dc9a95b01496760ebce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46201 } }
I20260928 05:16:15.954203 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:15.954329 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0139f1947564dc9a95b01496760ebce, State: Initialized, Role: FOLLOWER
I20260928 05:16:15.954972 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0139f1947564dc9a95b01496760ebce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46201 } }
I20260928 05:16:15.955197 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:15.955339 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:15.955513 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:15.958098 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0139f1947564dc9a95b01496760ebce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46201 } }
I20260928 05:16:15.958684 22413 leader_election.cc:302] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0139f1947564dc9a95b01496760ebce; no voters:
I20260928 05:16:15.959259 22413 leader_election.cc:288] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:15.959410 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:15.960075 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [term 1 LEADER]: Becoming Leader. State: Replica: c0139f1947564dc9a95b01496760ebce, State: Running, Role: LEADER
I20260928 05:16:15.960664 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0139f1947564dc9a95b01496760ebce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46201 } }
I20260928 05:16:15.961402 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:15.963759 22425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0139f1947564dc9a95b01496760ebce. Latest consensus state: current_term: 1 leader_uuid: "c0139f1947564dc9a95b01496760ebce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0139f1947564dc9a95b01496760ebce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46201 } } }
I20260928 05:16:15.964041 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0139f1947564dc9a95b01496760ebce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0139f1947564dc9a95b01496760ebce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46201 } } }
I20260928 05:16:15.964277 22425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:15.964324 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:15.965940 22432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:15.970654 22432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:15.979873 22432 catalog_manager.cc:1347] Generated new cluster ID: b4e437dfdbc346da9f7f46747b039e9f
I20260928 05:16:15.980013 22432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:15.998853 22432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:15.999361 22432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:16.007920 22432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0139f1947564dc9a95b01496760ebce: Generated new TSK 0
I20260928 05:16:16.009050 22432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:16.024022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22346
W20260928 05:16:16.025440 22408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46201
W20260928 05:16:16.026177 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:16.026357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22346
2026-09-28T05:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:16 dist-test-slave-6718 krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:16 dist-test-slave-6718 krb5kdc[22444](info): set up 2 sockets
Sep 28 05:16:16 dist-test-slave-6718 krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260928 05:16:17.122951 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:17 dist-test-slave-6718 krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:17.398025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44049
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:17.724668 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:17.724969 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:17.725142 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:17.732302 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:17.732410 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:17.732525 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:17.732627 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:17.732717 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:17.742615 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:17.743908 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:17.745213 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:17.756974 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:17.756896 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:17.756934 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:17.757969 22457 server_base.cc:1034] running on GCE node
I20260928 05:16:17.758875 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:17.760360 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:17.761570 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790572577761525 us; error 38 us; skew 500 ppm
I20260928 05:16:17.761898 22457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:17 dist-test-slave-6718 krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:17.765064 22457 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:17.766703 22457 webserver.cc:460] Webserver started at http://127.0.97.62:37555/ using document root <none> and password file <none>
I20260928 05:16:17.767421 22457 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:17.767597 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:17.767926 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:17.771064 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "9a513991dcb9427cb1f0a36f46264af5"
format_stamp: "Formatted at 2026-09-28 05:16:17 on dist-test-slave-6718"
server_key: "4b346030ff1f96f37ba1030ed7815afc"
server_key_iv: "c22a925378282ec59fbc967ea3a33ee0"
server_key_version: "encryptionkey@0"
I20260928 05:16:17.771870 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a513991dcb9427cb1f0a36f46264af5"
format_stamp: "Formatted at 2026-09-28 05:16:17 on dist-test-slave-6718"
server_key: "4b346030ff1f96f37ba1030ed7815afc"
server_key_iv: "c22a925378282ec59fbc967ea3a33ee0"
server_key_version: "encryptionkey@0"
I20260928 05:16:17.777040 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:16:17.780450 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:17.781719 22457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 05:16:17.781903 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "9a513991dcb9427cb1f0a36f46264af5"
format_stamp: "Formatted at 2026-09-28 05:16:17 on dist-test-slave-6718"
server_key: "4b346030ff1f96f37ba1030ed7815afc"
server_key_iv: "c22a925378282ec59fbc967ea3a33ee0"
server_key_version: "encryptionkey@0"
I20260928 05:16:17.782063 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:17.799523 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:17.803597 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:17.803897 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:17.822588 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44049
I20260928 05:16:17.822657 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44049 every 8 connection(s)
I20260928 05:16:17.823990 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:17.827668 22524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:17.832650 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22457
I20260928 05:16:17.832854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:17.833340 388 external_mini_cluster.cc:1428] Setting key 611e4a1ad535bcd9518b2924fdab70d6
I20260928 05:16:17.839324 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5: Bootstrap starting.
I20260928 05:16:17.843123 22524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:17.844333 22524 log.cc:824] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:17.847944 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5: No bootstrap required, opened a new log
Sep 28 05:16:17 dist-test-slave-6718 krb5kdc[22444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:16:17.855790 22524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a513991dcb9427cb1f0a36f46264af5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260928 05:16:17.856284 22524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:17.856406 22524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a513991dcb9427cb1f0a36f46264af5, State: Initialized, Role: FOLLOWER
I20260928 05:16:17.857020 22524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a513991dcb9427cb1f0a36f46264af5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260928 05:16:17.857264 22524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:17.857416 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:17.835143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52235 (local address 127.0.97.62:44049)
0928 05:16:17.835988 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:17.836000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:17.836052 (+ 52us) server_negotiation.cc:407] Connection header received
0928 05:16:17.837852 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:17.837875 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:17.838299 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:17.838706 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:17.840500 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:17.841839 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:17.843631 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:17.844317 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:17.844450 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:17.849513 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:17.849521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:17.849539 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:17.849578 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:17.852855 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:17.853581 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:17.853588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:17.853594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:17.853764 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:17.854208 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:17.854212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:17.854217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:17.854877 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:17.855210 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:17.855624 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 05:16:17.856082 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":398,"threads_started":1}
I20260928 05:16:17.857440 22524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:17.857683 22524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:17.860164 22524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a513991dcb9427cb1f0a36f46264af5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260928 05:16:17.860745 22524 leader_election.cc:302] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a513991dcb9427cb1f0a36f46264af5; no voters:
I20260928 05:16:17.861336 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:17.861397 22524 leader_election.cc:288] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:17.861526 22529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:17.864002 22524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:17.868170 22529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [term 1 LEADER]: Becoming Leader. State: Replica: 9a513991dcb9427cb1f0a36f46264af5, State: Running, Role: LEADER
I20260928 05:16:17.868836 22529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a513991dcb9427cb1f0a36f46264af5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260928 05:16:17.872377 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a513991dcb9427cb1f0a36f46264af5. Latest consensus state: current_term: 1 leader_uuid: "9a513991dcb9427cb1f0a36f46264af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a513991dcb9427cb1f0a36f46264af5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } } }
I20260928 05:16:17.872804 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:17.873886 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a513991dcb9427cb1f0a36f46264af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a513991dcb9427cb1f0a36f46264af5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } } }
I20260928 05:16:17.874224 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:17.885100 22544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:17.890656 22544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:17.903370 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:17.873516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48547 (local address 127.0.97.62:44049)
0928 05:16:17.873763 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:17.873772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:17.884060 (+ 10288us) server_negotiation.cc:407] Connection header received
0928 05:16:17.884379 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:17.884385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:17.884469 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:17.884560 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:17.886942 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:17.888033 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:17.889271 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:17.889967 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:17.890097 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:17.893996 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:17.894006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:17.894014 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:17.894067 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:17.898958 (+ 4891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:17.900088 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:17.900097 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:17.900103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:17.900301 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:17.900843 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:17.900851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:17.900857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:17.901441 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:17.901752 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:17.902498 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 05:16:17.902816 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 05:16:17.911475 22544 catalog_manager.cc:1347] Generated new cluster ID: 59577625155b475ea40f6cdf37c804f3
I20260928 05:16:17.911599 22544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:17.931399 22544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:17.932766 22544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:17.940117 22544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a513991dcb9427cb1f0a36f46264af5: Generated new TSK 0
I20260928 05:16:17.941030 22544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:17.947976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22457
I20260928 05:16:17.949069 22519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:18.040786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22457
2026-09-28T05:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:18 dist-test-slave-6718 krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:18 dist-test-slave-6718 krb5kdc[22556](info): set up 2 sockets
Sep 28 05:16:18 dist-test-slave-6718 krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260928 05:16:19.139034 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:19 dist-test-slave-6718 krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:19.395678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34161
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:19.723599 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:19.723905 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:19.723999 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:19.731284 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:19.731410 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:19.731456 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:19.731509 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:19.731554 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:19.741734 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:19.743176 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:19.744411 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:19.756316 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:19.756239 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:19.756388 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:19.757258 22569 server_base.cc:1034] running on GCE node
I20260928 05:16:19.758040 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:19.759555 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:19.760790 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790572579760730 us; error 32 us; skew 500 ppm
I20260928 05:16:19.761157 22569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:19 dist-test-slave-6718 krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:19.764430 22569 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:19.766116 22569 webserver.cc:460] Webserver started at http://127.0.97.62:35735/ using document root <none> and password file <none>
I20260928 05:16:19.766927 22569 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:19.767059 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:19.767385 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:19.770357 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "8fa228b280b4464e93584816dad5631e"
format_stamp: "Formatted at 2026-09-28 05:16:19 on dist-test-slave-6718"
server_key: "1502e7b981e13887e5ccca6ac43c6dc9"
server_key_iv: "fdb67e35160a243816946099389a8c3b"
server_key_version: "encryptionkey@0"
I20260928 05:16:19.771327 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "8fa228b280b4464e93584816dad5631e"
format_stamp: "Formatted at 2026-09-28 05:16:19 on dist-test-slave-6718"
server_key: "1502e7b981e13887e5ccca6ac43c6dc9"
server_key_iv: "fdb67e35160a243816946099389a8c3b"
server_key_version: "encryptionkey@0"
I20260928 05:16:19.776861 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:16:19.780616 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:19.782174 22569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 05:16:19.782356 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "8fa228b280b4464e93584816dad5631e"
format_stamp: "Formatted at 2026-09-28 05:16:19 on dist-test-slave-6718"
server_key: "1502e7b981e13887e5ccca6ac43c6dc9"
server_key_iv: "fdb67e35160a243816946099389a8c3b"
server_key_version: "encryptionkey@0"
I20260928 05:16:19.782558 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:19.796710 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:19.800895 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:19.801126 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:19.819916 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34161
I20260928 05:16:19.820017 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34161 every 8 connection(s)
I20260928 05:16:19.821326 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:19.825073 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:19.829808 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22569
I20260928 05:16:19.829993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:19.830559 388 external_mini_cluster.cc:1428] Setting key 3f28cd93abcb12adcfe6e040ee1647e3
I20260928 05:16:19.836498 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e: Bootstrap starting.
I20260928 05:16:19.840222 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:19.841657 22636 log.cc:824] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:19.844559 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:19.832282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41503 (local address 127.0.97.62:34161)
0928 05:16:19.833063 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:19.833074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:19.833116 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:16:19.835009 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:19.835033 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:19.835424 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:19.835761 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:19.836329 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:19.837744 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:19.839694 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:19.840447 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:19.840588 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:19.841909 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:19.841917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:19.841933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:19.842734 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:19.842838 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:19.842882 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:19.843014 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:19.843243 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 05:16:19.843546 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":347,"threads_started":1}
I20260928 05:16:19.845259 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e: No bootstrap required, opened a new log
I20260928 05:16:19.848506 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:19.855104 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa228b280b4464e93584816dad5631e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34161 } }
I20260928 05:16:19.855631 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:19.855752 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa228b280b4464e93584816dad5631e, State: Initialized, Role: FOLLOWER
I20260928 05:16:19.856511 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa228b280b4464e93584816dad5631e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34161 } }
I20260928 05:16:19.856815 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:19.856963 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:19.857169 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:19.859949 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:19.852926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37001 (local address 127.0.97.62:34161)
0928 05:16:19.853139 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:19.853145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:19.853498 (+ 353us) server_negotiation.cc:407] Connection header received
0928 05:16:19.853693 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:19.853697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:19.853765 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:19.853855 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:19.854481 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:19.855269 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:19.857353 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:19.857785 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:19.857911 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:19.859198 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:19.859203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:19.859206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:19.859224 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:19.859268 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:19.859283 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:19.859356 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:19.859532 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 05:16:19.859605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 05:16:19.860760 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa228b280b4464e93584816dad5631e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34161 } }
I20260928 05:16:19.861446 22636 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fa228b280b4464e93584816dad5631e; no voters:
I20260928 05:16:19.862149 22636 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:19.862257 22646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:19.863045 22646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [term 1 LEADER]: Becoming Leader. State: Replica: 8fa228b280b4464e93584816dad5631e, State: Running, Role: LEADER
I20260928 05:16:19.863620 22646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa228b280b4464e93584816dad5631e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34161 } }
I20260928 05:16:19.864602 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:19.866788 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fa228b280b4464e93584816dad5631e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa228b280b4464e93584816dad5631e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34161 } } }
I20260928 05:16:19.867312 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:19.868170 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa228b280b4464e93584816dad5631e. Latest consensus state: current_term: 1 leader_uuid: "8fa228b280b4464e93584816dad5631e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa228b280b4464e93584816dad5631e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34161 } } }
I20260928 05:16:19.868593 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:19.868935 22654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:19.874081 22654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:19.884400 22654 catalog_manager.cc:1347] Generated new cluster ID: 41031b7677f0481d87368b36193e53e8
I20260928 05:16:19.884531 22654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:19.930490 22654 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:19.931882 22654 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:19.945511 22654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fa228b280b4464e93584816dad5631e: Generated new TSK 0
I20260928 05:16:19.946359 22654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:19.955232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22569
W20260928 05:16:19.956387 22631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41503
W20260928 05:16:19.957091 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:19.957240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22569
2026-09-28T05:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:20 dist-test-slave-6718 krb5kdc[22667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:20 dist-test-slave-6718 krb5kdc[22667](info): set up 2 sockets
Sep 28 05:16:20 dist-test-slave-6718 krb5kdc[22667](info): commencing operation
krb5kdc: starting...
W20260928 05:16:21.071127 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:21 dist-test-slave-6718 krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:21.330417 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:21.663721 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:21.664013 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:21.664178 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:21.671407 22680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:21.671515 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:21.671609 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:21.671712 22680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:21.671802 22680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:21.681640 22680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:21.682948 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:21.684224 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:21.695911 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:21.695943 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:21.696053 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:21.696646 22680 server_base.cc:1034] running on GCE node
I20260928 05:16:21.697412 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:21.698901 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:21.700127 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790572581700086 us; error 34 us; skew 500 ppm
I20260928 05:16:21.700474 22680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:21 dist-test-slave-6718 krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:21.708621 22680 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:21.710245 22680 webserver.cc:460] Webserver started at http://127.0.97.62:33655/ using document root <none> and password file <none>
I20260928 05:16:21.711025 22680 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:21.711171 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:21.711485 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:21.714473 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "184ea20aa0b44d979e06c6bf59db062f"
format_stamp: "Formatted at 2026-09-28 05:16:21 on dist-test-slave-6718"
server_key: "b734bc5de60cc3954d9847c37585b6f7"
server_key_iv: "6ad342aebd125ae7379a3e5df49300d0"
server_key_version: "encryptionkey@0"
I20260928 05:16:21.715304 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "184ea20aa0b44d979e06c6bf59db062f"
format_stamp: "Formatted at 2026-09-28 05:16:21 on dist-test-slave-6718"
server_key: "b734bc5de60cc3954d9847c37585b6f7"
server_key_iv: "6ad342aebd125ae7379a3e5df49300d0"
server_key_version: "encryptionkey@0"
I20260928 05:16:21.720566 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 05:16:21.724190 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:21.725589 22680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 05:16:21.725777 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "184ea20aa0b44d979e06c6bf59db062f"
format_stamp: "Formatted at 2026-09-28 05:16:21 on dist-test-slave-6718"
server_key: "b734bc5de60cc3954d9847c37585b6f7"
server_key_iv: "6ad342aebd125ae7379a3e5df49300d0"
server_key_version: "encryptionkey@0"
I20260928 05:16:21.725982 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:21.731626 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:21.735553 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:21.735848 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:21.754791 22680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35823
I20260928 05:16:21.754845 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35823 every 8 connection(s)
I20260928 05:16:21.756245 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:21.760018 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:21.765512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22680
I20260928 05:16:21.765688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:21.766135 388 external_mini_cluster.cc:1428] Setting key 9d1e9677cc26e9bf67b26de95faf9cdd
I20260928 05:16:21.771035 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f: Bootstrap starting.
I20260928 05:16:21.773975 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:21.767919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54327 (local address 127.0.97.62:35823)
0928 05:16:21.768858 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:21.768870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:21.768911 (+ 41us) server_negotiation.cc:407] Connection header received
0928 05:16:21.770648 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:21.770676 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:21.771040 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:21.771412 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:21.772460 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:21.772466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:21.772479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:21.772495 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:21.772550 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:21.772569 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:21.772743 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:21.772884 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 05:16:21.773094 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":478,"threads_started":1}
I20260928 05:16:21.774731 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:21.775893 22747 log.cc:824] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:21.777974 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:21.778846 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f: No bootstrap required, opened a new log
I20260928 05:16:21.784971 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:21.782358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47533 (local address 127.0.97.62:35823)
0928 05:16:21.782608 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:21.782619 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:21.782935 (+ 316us) server_negotiation.cc:407] Connection header received
0928 05:16:21.783101 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:21.783105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:21.783149 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:21.783238 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:21.784216 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:21.784220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:21.784224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 05:16:21.784237 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:21.784281 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 05:16:21.784295 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:21.784388 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:21.784563 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 05:16:21.784615 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 05:16:21.787104 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184ea20aa0b44d979e06c6bf59db062f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35823 } }
I20260928 05:16:21.787463 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 05:16:21.787561 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184ea20aa0b44d979e06c6bf59db062f, State: Initialized, Role: FOLLOWER
I20260928 05:16:21.788237 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184ea20aa0b44d979e06c6bf59db062f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35823 } }
I20260928 05:16:21.788532 22747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 05:16:21.788671 22747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 05:16:21.788870 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:21.791072 22747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184ea20aa0b44d979e06c6bf59db062f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35823 } }
I20260928 05:16:21.791549 22747 leader_election.cc:302] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184ea20aa0b44d979e06c6bf59db062f; no voters:
I20260928 05:16:21.792197 22747 leader_election.cc:288] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 05:16:21.792316 22757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:21.793036 22757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [term 1 LEADER]: Becoming Leader. State: Replica: 184ea20aa0b44d979e06c6bf59db062f, State: Running, Role: LEADER
I20260928 05:16:21.793620 22757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184ea20aa0b44d979e06c6bf59db062f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35823 } }
I20260928 05:16:21.794440 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:21.797382 22759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184ea20aa0b44d979e06c6bf59db062f. Latest consensus state: current_term: 1 leader_uuid: "184ea20aa0b44d979e06c6bf59db062f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184ea20aa0b44d979e06c6bf59db062f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35823 } } }
I20260928 05:16:21.797736 22759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:21.798645 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "184ea20aa0b44d979e06c6bf59db062f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184ea20aa0b44d979e06c6bf59db062f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35823 } } }
I20260928 05:16:21.799018 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:21.799422 22763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:21.805838 22763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:21.814890 22763 catalog_manager.cc:1347] Generated new cluster ID: 2a6b48ac5c234065a2f3400befa31520
I20260928 05:16:21.815030 22763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:21.828745 22763 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:21.829234 22763 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:21.838088 22763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184ea20aa0b44d979e06c6bf59db062f: Generated new TSK 0
I20260928 05:16:21.838964 22763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:21.848574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22680
W20260928 05:16:21.850136 22742 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54327
W20260928 05:16:21.850890 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 05:16:21.851045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22680
2026-09-28T05:16:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1893 ms)
[----------] 18 tests from AuthTokenIssuingTest (34922 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:21 dist-test-slave-6718 krb5kdc[22778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:21 dist-test-slave-6718 krb5kdc[22778](info): set up 2 sockets
Sep 28 05:16:21 dist-test-slave-6718 krb5kdc[22778](info): commencing operation
krb5kdc: starting...
W20260928 05:16:22.959261 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:23 dist-test-slave-6718 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:23.216013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45745
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45745,127.0.97.61:46337,127.0.97.60:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:23.551097 22791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:16:23.551514 22791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:23.551645 22791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:23.551792 22791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:23.559317 22791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:23.559455 22791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:23.559562 22791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:23.559650 22791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:23.559746 22791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:23.570019 22791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45745,127.0.97.61:46337,127.0.97.60:34109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:23.571439 22791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:23.572784 22791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:23.584411 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:23.584374 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:23.584470 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:23.585215 22791 server_base.cc:1034] running on GCE node
I20260928 05:16:23.586015 22791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:23.587472 22791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:23.588716 22791 hybrid_clock.cc:648] HybridClock initialized: now 1790572583588662 us; error 32 us; skew 500 ppm
I20260928 05:16:23.589051 22791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:23 dist-test-slave-6718 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:23.592272 22791 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:23.593739 22791 webserver.cc:460] Webserver started at http://127.0.97.62:45419/ using document root <none> and password file <none>
I20260928 05:16:23.594436 22791 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:23.594594 22791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:23.594937 22791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:23.597895 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "839426d6a88944a1a2f9a75ba4d3254b"
format_stamp: "Formatted at 2026-09-28 05:16:23 on dist-test-slave-6718"
server_key: "36cc33f4b7ebee02f7a75299d375d3dc"
server_key_iv: "41be76810e005913d178ee54c6d118ec"
server_key_version: "encryptionkey@0"
I20260928 05:16:23.598838 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "839426d6a88944a1a2f9a75ba4d3254b"
format_stamp: "Formatted at 2026-09-28 05:16:23 on dist-test-slave-6718"
server_key: "36cc33f4b7ebee02f7a75299d375d3dc"
server_key_iv: "41be76810e005913d178ee54c6d118ec"
server_key_version: "encryptionkey@0"
I20260928 05:16:23.604254 22791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 05:16:23.607786 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:23.609241 22791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 05:16:23.609452 22791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "839426d6a88944a1a2f9a75ba4d3254b"
format_stamp: "Formatted at 2026-09-28 05:16:23 on dist-test-slave-6718"
server_key: "36cc33f4b7ebee02f7a75299d375d3dc"
server_key_iv: "41be76810e005913d178ee54c6d118ec"
server_key_version: "encryptionkey@0"
I20260928 05:16:23.609687 22791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:23.624352 22791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:23.628859 22791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:23.629143 22791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:23.648355 22791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45745
I20260928 05:16:23.648396 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45745 every 8 connection(s)
I20260928 05:16:23.649785 22791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:23.651594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22791
I20260928 05:16:23.651822 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:23.652624 388 external_mini_cluster.cc:1428] Setting key 1ce619de9dc1c428dd8d78b3f95ff9f6
I20260928 05:16:23.654656 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:23.664240 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 28 05:16:23 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:16:23.691584 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.667249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45745 (local address 127.0.0.1:46090)
0928 05:16:23.668070 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.668119 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:23.670157 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:23.671168 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:23.671174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:23.671723 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:23.672073 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:23.672091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:23.674413 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:23.674416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:23.675662 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:23.675674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:23.677075 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:23.677079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:23.677220 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:23.677952 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:23.677975 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:23.682623 (+ 4648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:23.686972 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:23.686986 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:23.687002 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:23.687530 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:23.688020 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:23.688027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:23.688033 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:23.688220 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:23.689208 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:23.689218 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:23.689457 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 05:16:23.689707 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 05:16:23.690243 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260928 05:16:23.691682 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.668563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.0.97.62:45745)
0928 05:16:23.669269 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:23.669280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:23.669331 (+ 51us) server_negotiation.cc:407] Connection header received
0928 05:16:23.670558 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:23.670568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:23.670911 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:23.671118 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:23.672288 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:23.674256 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:23.675812 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:23.676864 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:23.677056 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:23.682870 (+ 5814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:23.682885 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:23.682905 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:23.682950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:23.686819 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:23.687693 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:23.687705 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:23.687711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:23.687903 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:23.688356 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:23.688363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:23.688369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:23.689075 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:23.689282 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:23.689894 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 05:16:23.690427 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":493,"thread_start_us":324,"threads_started":1}
I20260928 05:16:23.694728 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:16:23.695925 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.695414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60266)
0928 05:16:23.695641 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.695775 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 05:16:23.696239 22808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
I20260928 05:16:23.697887 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.697428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60272)
0928 05:16:23.697611 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.697714 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 05:16:23.698096 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46337: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 05:16:23.715826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46337
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45745,127.0.97.61:46337,127.0.97.60:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:16:23.747445 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } attempt: 1
I20260928 05:16:23.748723 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.748201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60278)
0928 05:16:23.748432 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.748544 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 05:16:23.749820 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.749466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60292)
0928 05:16:23.749603 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.749667 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 05:16:23.750082 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46337: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
I20260928 05:16:23.818358 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } attempt: 2
I20260928 05:16:23.819519 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.819068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60296)
0928 05:16:23.819268 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.819382 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 05:16:23.820510 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.820156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60306)
0928 05:16:23.820312 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.820388 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 05:16:23.820688 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46337: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
I20260928 05:16:23.911949 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } attempt: 3
I20260928 05:16:23.913175 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.912699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60314)
0928 05:16:23.912929 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.913045 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 05:16:23.914155 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:23.913810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60318)
0928 05:16:23.913956 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:23.914029 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 05:16:23.914312 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46337: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
W20260928 05:16:24.042845 22865 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:16:24.043203 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:24.043334 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:24.043464 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:24.050639 22865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:24.050725 22865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:24.050832 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:24.050931 22865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:24.051020 22865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:24.057597 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } attempt: 4
I20260928 05:16:24.058921 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.058267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60322)
0928 05:16:24.058608 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.058741 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 05:16:24.059870 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.059537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60332)
0928 05:16:24.059656 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.059732 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 05:16:24.060070 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46337: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46337: connect: Connection refused (error 111)
I20260928 05:16:24.061458 22865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45745,127.0.97.61:46337,127.0.97.60:34109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:24.062786 22865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:24.064073 22865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:24.075361 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:24.075367 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:24.075438 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:24.075929 22865 server_base.cc:1034] running on GCE node
I20260928 05:16:24.076745 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:24.078088 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:24.079336 22865 hybrid_clock.cc:648] HybridClock initialized: now 1790572584079260 us; error 62 us; skew 500 ppm
I20260928 05:16:24.079662 22865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:24.082741 22865 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 05:16:24.084088 22865 webserver.cc:460] Webserver started at http://127.0.97.61:34623/ using document root <none> and password file <none>
I20260928 05:16:24.084791 22865 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:24.084921 22865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:24.085233 22865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:24.088178 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data/instance:
uuid: "2553fe98df544a7e85a4a5cbdfc779c9"
format_stamp: "Formatted at 2026-09-28 05:16:24 on dist-test-slave-6718"
server_key: "456dbc1e4d0f99d4e6d5ddddff65e170"
server_key_iv: "3ac173eac16476362edc5877ae521ee2"
server_key_version: "encryptionkey@0"
I20260928 05:16:24.088971 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal/instance:
uuid: "2553fe98df544a7e85a4a5cbdfc779c9"
format_stamp: "Formatted at 2026-09-28 05:16:24 on dist-test-slave-6718"
server_key: "456dbc1e4d0f99d4e6d5ddddff65e170"
server_key_iv: "3ac173eac16476362edc5877ae521ee2"
server_key_version: "encryptionkey@0"
I20260928 05:16:24.093978 22865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 05:16:24.097354 22880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:24.098748 22865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 05:16:24.098927 22865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal
uuid: "2553fe98df544a7e85a4a5cbdfc779c9"
format_stamp: "Formatted at 2026-09-28 05:16:24 on dist-test-slave-6718"
server_key: "456dbc1e4d0f99d4e6d5ddddff65e170"
server_key_iv: "3ac173eac16476362edc5877ae521ee2"
server_key_version: "encryptionkey@0"
I20260928 05:16:24.099143 22865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:24.114976 22865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:24.119278 22865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:24.119558 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:24.137434 22865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46337
I20260928 05:16:24.137454 22931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46337 every 8 connection(s)
I20260928 05:16:24.138852 22865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:16:24.140491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22865
I20260928 05:16:24.140682 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-1/wal/instance
I20260928 05:16:24.141250 388 external_mini_cluster.cc:1428] Setting key 6f4796346725b3feccfff7f7d54fcb5a
I20260928 05:16:24.143504 22932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:24.152740 22932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260928 05:16:24.175457 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.155405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.97.62:45745)
0928 05:16:24.155646 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.155655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.156743 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 05:16:24.159186 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.159195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.159295 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.159419 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.160918 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.162096 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.163938 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.164735 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.164905 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.168644 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.168650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.168653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.168688 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.171773 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.172860 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.172865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.172867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.172998 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.173532 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.173537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.173540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.173898 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.174024 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.175030 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.175211 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 05:16:24.177042 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.155747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45745 (local address 127.0.0.1:46094)
0928 05:16:24.156565 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.156611 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.158891 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.159614 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.159622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.160139 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.160710 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.160745 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.162298 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.162302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.163764 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.163781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.164996 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.165001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.165213 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.165952 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.165974 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.168387 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.171960 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.171972 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.172000 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.172684 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.173120 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.173127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.173133 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.173376 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.174044 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.174055 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.174582 (+ 527us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.174910 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.175500 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":357,"threads_started":1}
I20260928 05:16:24.178987 22932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:24.194159 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.179918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60346)
0928 05:16:24.180116 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.180136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.180253 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.181481 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.181485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.181924 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.182050 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.182058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.183479 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.183487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.184316 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.184327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.185138 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.185143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.185324 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.185911 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.185927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.188312 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.191682 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.191690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.191696 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.192047 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.192406 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.192415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.192419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.192554 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.193190 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.193194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.193332 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.193489 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.193777 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 05:16:24.194268 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.179918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.97.61:46337)
0928 05:16:24.181075 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.181081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.181105 (+ 24us) server_negotiation.cc:407] Connection header received
0928 05:16:24.181146 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.181152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.181326 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.181501 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.182226 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.183336 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.184449 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.184977 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.185122 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.188556 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.188568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.188583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.188610 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.191549 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.192179 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.192193 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.192196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.192299 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.192655 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.192663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.192666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.193049 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.193267 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.193633 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.193904 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1025,"thread_start_us":896,"threads_started":1}
I20260928 05:16:24.196887 22932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:16:24.198113 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.197675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52226)
0928 05:16:24.197854 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.197970 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 05:16:24.198415 22881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.199980 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.199601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52230)
0928 05:16:24.199805 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.199882 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 05:16:24.200281 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 05:16:24.202198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34109
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45745,127.0.97.61:46337,127.0.97.60:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:16:24.249609 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 1
I20260928 05:16:24.250998 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.250448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52240)
0928 05:16:24.250718 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.250847 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 05:16:24.252009 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.251659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52254)
0928 05:16:24.251813 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.251905 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 05:16:24.252203 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.320418 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 2
I20260928 05:16:24.321658 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.321077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52258)
0928 05:16:24.321313 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.321440 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 05:16:24.323045 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.322518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52262)
0928 05:16:24.322761 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.322868 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 05:16:24.323324 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.359340 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } attempt: 5
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:24.373426 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.360191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60362)
0928 05:16:24.360421 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.360443 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.360591 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.361005 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.361010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.361428 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.361602 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.361612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.362745 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.362756 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.363619 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.363631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.364368 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.364374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.364548 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.365194 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.365210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.367301 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.371034 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.371038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.371041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.371399 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.371839 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.371844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.371846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.371998 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.372679 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.372683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.372830 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.372946 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.373173 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 05:16:24.373737 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.360242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.0.97.61:46337)
0928 05:16:24.360459 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.360467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.360512 (+ 45us) server_negotiation.cc:407] Connection header received
0928 05:16:24.360749 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.360753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.360857 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.360971 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.361770 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.362600 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.363770 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.364206 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.364336 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.367502 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.367509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.367514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.367556 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.370883 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.371509 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.371514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.371517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.371664 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.372160 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.372166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.372172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.372554 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.372708 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.373084 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.373336 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 05:16:24.374825 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:16:24.376164 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.375607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52278)
0928 05:16:24.375839 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.375953 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:16:24.377203 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.376820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52294)
0928 05:16:24.376972 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.377059 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:16:24.377475 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.414614 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 3
I20260928 05:16:24.415959 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.415289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52302)
0928 05:16:24.415643 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.415759 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260928 05:16:24.417346 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.416756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52304)
0928 05:16:24.417052 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.417167 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260928 05:16:24.417668 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.428726 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 1
I20260928 05:16:24.429934 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.429363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52316)
0928 05:16:24.429607 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.429731 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:16:24.431005 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.430656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52330)
0928 05:16:24.430798 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.430881 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 05:16:24.431217 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.499439 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 2
I20260928 05:16:24.500765 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.500137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52344)
0928 05:16:24.500386 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.500532 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:16:24.501869 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.501484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52360)
0928 05:16:24.501634 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.501726 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:16:24.502086 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
W20260928 05:16:24.536302 22939 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:16:24.536616 22939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:24.536689 22939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:24.536772 22939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:24.543896 22939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:24.543990 22939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:24.544046 22939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:24.544097 22939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:24.544159 22939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:24.553867 22939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45745,127.0.97.61:46337,127.0.97.60:34109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:24.555178 22939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:24.556464 22939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 05:16:24.560938 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 4
I20260928 05:16:24.562314 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.561609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52366)
0928 05:16:24.561974 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.562099 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 05:16:24.563536 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.563116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52374)
0928 05:16:24.563293 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.563393 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 05:16:24.563771 22932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
W20260928 05:16:24.568689 22947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:24.568775 22945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:24.568750 22944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:24.569571 22939 server_base.cc:1034] running on GCE node
I20260928 05:16:24.570359 22939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:24.571851 22939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:24.573104 22939 hybrid_clock.cc:648] HybridClock initialized: now 1790572584573031 us; error 50 us; skew 500 ppm
I20260928 05:16:24.573444 22939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:24.576612 22939 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 05:16:24.578099 22939 webserver.cc:460] Webserver started at http://127.0.97.60:42467/ using document root <none> and password file <none>
I20260928 05:16:24.578895 22939 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:24.579026 22939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:24.579360 22939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:24.582295 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data/instance:
uuid: "65c37adac3704fd9812201ccee5a9df3"
format_stamp: "Formatted at 2026-09-28 05:16:24 on dist-test-slave-6718"
server_key: "dc6452f77f73d0e4c654786a1e6d4be0"
server_key_iv: "a898e00061894868c32dcbaadc43b6d3"
server_key_version: "encryptionkey@0"
I20260928 05:16:24.583175 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal/instance:
uuid: "65c37adac3704fd9812201ccee5a9df3"
format_stamp: "Formatted at 2026-09-28 05:16:24 on dist-test-slave-6718"
server_key: "dc6452f77f73d0e4c654786a1e6d4be0"
server_key_iv: "a898e00061894868c32dcbaadc43b6d3"
server_key_version: "encryptionkey@0"
I20260928 05:16:24.588332 22939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:16:24.591769 22954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:24.593116 22939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:16:24.593303 22939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal
uuid: "65c37adac3704fd9812201ccee5a9df3"
format_stamp: "Formatted at 2026-09-28 05:16:24 on dist-test-slave-6718"
server_key: "dc6452f77f73d0e4c654786a1e6d4be0"
server_key_iv: "a898e00061894868c32dcbaadc43b6d3"
server_key_version: "encryptionkey@0"
I20260928 05:16:24.593550 22939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:24.608289 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 3
I20260928 05:16:24.609405 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.608909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52378)
0928 05:16:24.609130 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.609229 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 05:16:24.610452 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.610055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52384)
0928 05:16:24.610231 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.610329 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 05:16:24.610782 22858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34109: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34109: connect: Connection refused (error 111)
I20260928 05:16:24.620540 22939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:24.624874 22939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:24.625150 22939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:24.644219 22939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34109
I20260928 05:16:24.644232 23005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34109 every 8 connection(s)
I20260928 05:16:24.645583 22939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:16:24.647027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 22939
I20260928 05:16:24.647202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790572228671169-388-0/minicluster-data/master-2/wal/instance
I20260928 05:16:24.647734 388 external_mini_cluster.cc:1428] Setting key f64e78dd5559faceec7e5240344761ca
I20260928 05:16:24.650355 23006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:24.662993 23006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:16:24.682786 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.665830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.97.62:45745)
0928 05:16:24.666082 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.666090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.666610 (+ 520us) server_negotiation.cc:407] Connection header received
0928 05:16:24.666874 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.666880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.666966 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.667082 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.668399 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.669081 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.670687 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.671367 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.671495 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.675809 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.675815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.675818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.675852 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.679184 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.680063 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.680067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.680069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.680198 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.680666 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.680670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.680673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.681059 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.681225 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.681928 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.682176 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:16:24.683554 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.665783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45745 (local address 127.0.0.1:46108)
0928 05:16:24.666465 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.666512 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.666729 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.667135 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.667140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.667699 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.668207 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.668232 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.669241 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.669246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.670472 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.670483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.671600 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.671605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.671822 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.672623 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.672663 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.675595 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.679315 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.679324 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.679333 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.679906 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.680308 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.680315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.680321 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.680550 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.681206 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.681212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.681518 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.681801 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.682337 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":55,"thread_start_us":288,"threads_started":1}
I20260928 05:16:24.683571 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.649576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.0.97.60:34109)
0928 05:16:24.663177 (+ 13601us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.663194 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.663569 (+ 375us) server_negotiation.cc:407] Connection header received
0928 05:16:24.665496 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.665517 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.666054 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.666511 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.668044 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.669415 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.670835 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.671695 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.671816 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.676395 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.676403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.676414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.676445 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.679874 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.680630 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.680641 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.680647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.680810 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.681167 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.681172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.681175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.681545 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.681785 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.682146 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.682465 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":13271,"spinlock_wait_cycles":12160,"thread_start_us":13028,"threads_started":1}
I20260928 05:16:24.685580 23006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:16:24.699152 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.686338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46337 (local address 127.0.0.1:60366)
0928 05:16:24.686501 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.686521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.686678 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.687162 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.687167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.687616 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.687756 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.687764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.688736 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.688741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.689597 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.689607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.690299 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.690304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.690500 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.691102 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.691118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.693189 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.696725 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.696730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.696733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.697077 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.697522 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.697526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.697530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.697724 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.698229 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.698232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.698354 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.698510 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.698751 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:16:24.699919 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.686490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.97.61:46337)
0928 05:16:24.686710 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.686718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.686740 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:16:24.686803 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.686811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.686885 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.686986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.687903 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.688588 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.689739 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.690147 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.690286 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.693438 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.693442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.693446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.693476 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.696597 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.697202 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.697207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.697211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.697407 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.697826 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.697831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.697834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.698121 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.698268 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.699451 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.699617 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 05:16:24.700871 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.686958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.0.97.62:45745)
0928 05:16:24.687155 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.687160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.687175 (+ 15us) server_negotiation.cc:407] Connection header received
0928 05:16:24.687247 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.687252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.687320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.687424 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.688506 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.689213 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.690355 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.690991 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.691123 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.694935 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.694939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.694943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.694993 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.698040 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.698856 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.698862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.698866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.699002 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.699401 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.699407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.699411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.699750 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.699953 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.700349 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.700563 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 05:16:24.700958 23006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:24.715425 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.701957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.0.97.60:34109)
0928 05:16:24.702124 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.702129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.702145 (+ 16us) server_negotiation.cc:407] Connection header received
0928 05:16:24.702191 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.702196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.702269 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.702394 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.703243 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.704102 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.705316 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.705688 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.705781 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.709574 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.709579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.709582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.709609 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.712420 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.713150 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.713155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.713159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.713370 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.713800 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.713805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.713808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.714209 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.714371 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.714832 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.715133 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":55}
I20260928 05:16:24.715504 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.701720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52396)
0928 05:16:24.701909 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.701928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.702020 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.702407 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.702411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.702919 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.703108 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.703119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.704227 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.704234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.705163 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.705178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.705829 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.705835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.706051 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.706705 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.706722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.709386 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.712545 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.712552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.712558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.713040 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.713489 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.713493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.713519 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.713674 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.714333 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.714337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.714502 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.714720 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.715006 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 05:16:24.717433 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.702604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.97.61:46337)
0928 05:16:24.702801 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.702806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.702824 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:16:24.702931 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.702934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.702984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.703049 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.704304 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.704989 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.706450 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.706950 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.707081 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.711579 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.711585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.711591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.711626 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.714788 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.715635 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.715644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.715649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.715777 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.716219 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.716223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.716225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.716486 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.716779 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.717006 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.717162 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:16:24.721112 23006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: Bootstrap starting.
I20260928 05:16:24.722312 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:24.724753 23006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:24.725790 23006 log.cc:824] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:24.724994 22833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "839426d6a88944a1a2f9a75ba4d3254b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46118
I20260928 05:16:24.728722 23006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: No bootstrap required, opened a new log
I20260928 05:16:24.735935 23006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.736277 23006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c37adac3704fd9812201ccee5a9df3, State: Initialized, Role: FOLLOWER
I20260928 05:16:24.736722 23006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.737713 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:24.738103 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:24.738761 23006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:16:24.748108 23023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:16:24.748242 23023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:16:24.746299 22939 master_runner.cc:186] Error getting master registration for 127.0.97.62:45745: OK, instance_id { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" instance_seqno: 1790572583642950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:16:24.754175 22939 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:16:24.788028 22858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 4
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:16:24.805225 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.788979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.0.97.60:34109)
0928 05:16:24.789171 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.789177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.789202 (+ 25us) server_negotiation.cc:407] Connection header received
0928 05:16:24.789307 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.789313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.789393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.789501 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.790215 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.791391 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.793198 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.793961 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.794097 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.798013 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.798020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.798025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.798068 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.801953 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.802765 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.802772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.802775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.802917 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.803395 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.803402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.803425 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.803975 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.804130 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.804600 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.804849 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 05:16:24.805208 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.788835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52398)
0928 05:16:24.789060 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.789081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.789206 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.789538 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.789541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.789899 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.790066 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.790074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.791537 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.791549 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.793031 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.793049 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.794120 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.794127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.794348 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.795080 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.795106 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.797826 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.802117 (+ 4291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.802122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.802126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.802663 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.803077 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.803082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.803086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.803254 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.804115 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.804121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.804320 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.804493 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.804803 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 05:16:24.811508 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b: Bootstrap starting.
I20260928 05:16:24.815587 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:24.816862 22858 log.cc:824] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:24.819895 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b: No bootstrap required, opened a new log
I20260928 05:16:24.827543 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.827956 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839426d6a88944a1a2f9a75ba4d3254b, State: Initialized, Role: FOLLOWER
I20260928 05:16:24.828392 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.829357 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:24.829699 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:24.830454 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:24.830629 22833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 05:16:24.830837 22833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:24.834378 22833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.836835 22981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "839426d6a88944a1a2f9a75ba4d3254b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65c37adac3704fd9812201ccee5a9df3"
I20260928 05:16:24.837332 22981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:24.839255 22833 leader_election.cc:288] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [CANDIDATE]: Term 1 election: Requested vote from peers 2553fe98df544a7e85a4a5cbdfc779c9 (127.0.97.61:46337), 65c37adac3704fd9812201ccee5a9df3 (127.0.97.60:34109)
I20260928 05:16:24.840857 22981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 839426d6a88944a1a2f9a75ba4d3254b in term 1.
I20260928 05:16:24.841504 22807 leader_election.cc:302] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c37adac3704fd9812201ccee5a9df3, 839426d6a88944a1a2f9a75ba4d3254b; no voters:
I20260928 05:16:24.841964 23026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:24.852099 23026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [term 1 LEADER]: Becoming Leader. State: Replica: 839426d6a88944a1a2f9a75ba4d3254b, State: Running, Role: LEADER
I20260928 05:16:24.852751 23026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.857846 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839426d6a88944a1a2f9a75ba4d3254b. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:24.858732 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:24.862993 22932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } attempt: 5
I20260928 05:16:24.866418 23035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 05:16:24 dist-test-slave-6718 krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260928 05:16:24.877229 22960 tablet.cc:2369] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 05:16:24.881991 22812 tablet.cc:2369] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:16:24.882794 23035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 05:16:24.884708 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.863615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34109 (local address 127.0.0.1:52406)
0928 05:16:24.863988 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:24.864008 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:24.864137 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:24.865234 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:24.865244 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:24.865905 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:24.866173 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.866189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.868068 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.868077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.869892 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:24.869912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.871747 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.871753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:24.871967 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.872670 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:24.872696 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:24.876191 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:24.881100 (+ 4909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.881108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.881116 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.881764 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.882327 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:24.882332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:24.882336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:24.882561 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:24.883405 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:24.883413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:24.883614 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 05:16:24.883801 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 05:16:24.884181 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260928 05:16:24.886101 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.864367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.0.97.60:34109)
0928 05:16:24.864604 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.864612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.864639 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:16:24.864718 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.864727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.864866 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.865001 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.866417 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.867863 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.870099 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.871396 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.871535 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.876454 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.876464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.876472 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.876524 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.880907 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.881961 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.881970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.881975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.882167 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.882713 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.882720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.882725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.883170 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.883319 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.885216 (+ 1897us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.885595 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:16:24.893237 22932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: Bootstrap starting.
I20260928 05:16:24.898494 22932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:24.900117 22932 log.cc:824] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:24.900645 22981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [term 1 FOLLOWER]: Refusing update from remote peer 839426d6a88944a1a2f9a75ba4d3254b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:16:24.901753 23032 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:16:24.916728 22932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: No bootstrap required, opened a new log
I20260928 05:16:24.949585 22932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.950126 22932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2553fe98df544a7e85a4a5cbdfc779c9, State: Initialized, Role: FOLLOWER
I20260928 05:16:24.960242 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839426d6a88944a1a2f9a75ba4d3254b. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:24.960665 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:24.963351 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:24.963631 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:24.963925 23040 mvcc.cc:205] Tried to move back new op lower bound from 7334185307740192768 to 7334185307562889216. Current Snapshot: MvccSnapshot[applied={T|T < 7334185307740192768}]
I20260928 05:16:24.966884 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.911544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.0.97.62:45745)
0928 05:16:24.911759 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.911767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.921326 (+ 9559us) server_negotiation.cc:407] Connection header received
0928 05:16:24.922472 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.922492 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.922632 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.922725 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.930368 (+ 7643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.931851 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.938316 (+ 6465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.939405 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.939542 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.954421 (+ 14879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.954432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.954440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.954505 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.962839 (+ 8334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.963650 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.963659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.963662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.963822 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.964274 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.964282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.964285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.964711 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.965004 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.966184 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.966452 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:16:24.962389 22932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } }
I20260928 05:16:24.970908 23045 mvcc.cc:205] Tried to move back new op lower bound from 7334185307740192768 to 7334185307562889216. Current Snapshot: MvccSnapshot[applied={T|T < 7334185307740192768}]
I20260928 05:16:24.970999 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.918851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.0.97.61:46337)
0928 05:16:24.919225 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.919232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.926669 (+ 7437us) server_negotiation.cc:407] Connection header received
0928 05:16:24.927250 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.927257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.927354 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.927447 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.934444 (+ 6997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.935843 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.940910 (+ 5067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.942137 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.942260 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.953983 (+ 11723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.953992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.953997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.954041 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.961144 (+ 7103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.968230 (+ 7086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.968240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.968246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.968428 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.968957 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.968962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.968965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.969361 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.969633 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.970256 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.970549 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260928 05:16:24.971359 23035 catalog_manager.cc:1347] Generated new cluster ID: 09fadf81f5b84bcda7ee2be72496ce30
I20260928 05:16:24.971457 23035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:24.977805 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:24.978081 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:24.993878 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:24.919028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.0.97.60:34109)
0928 05:16:24.921187 (+ 2159us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:24.921193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:24.931679 (+ 10486us) server_negotiation.cc:407] Connection header received
0928 05:16:24.932389 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:24.932396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:24.932514 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:24.932590 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:24.940153 (+ 7563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.941254 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.944688 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:24.945383 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:24.945483 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:24.980998 (+ 35515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:24.981005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:24.981010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:24.981058 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:24.984900 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.991371 (+ 6471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.991378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.991381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.991535 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:24.992005 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:24.992011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:24.992014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:24.992401 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:24.992650 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:24.993227 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 05:16:24.993497 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2022}
I20260928 05:16:24.994778 22932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:24.994823 22907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "839426d6a88944a1a2f9a75ba4d3254b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2553fe98df544a7e85a4a5cbdfc779c9"
I20260928 05:16:24.995775 22906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:24.997996 23055 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:16:24.995643 23054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:25.004169 23054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:25.006343 22906 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [term 1 FOLLOWER]: Refusing update from remote peer 839426d6a88944a1a2f9a75ba4d3254b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:16:25.014672 23026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:16:25.017102 22907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 839426d6a88944a1a2f9a75ba4d3254b in term 1.
I20260928 05:16:25.017150 23035 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:25.019163 23035 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:25.009739 22791 master_runner.cc:186] Error getting master registration for 127.0.97.61:46337: OK, instance_id { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" instance_seqno: 1790572584132404 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:16:25.024663 22791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:16:25.045202 23035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b: Generated new TSK 0
I20260928 05:16:25.046500 23035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:25.071687 23054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839426d6a88944a1a2f9a75ba4d3254b. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:25.072019 23054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:25.073418 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:25.073740 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839426d6a88944a1a2f9a75ba4d3254b [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:25.088634 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "839426d6a88944a1a2f9a75ba4d3254b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839426d6a88944a1a2f9a75ba4d3254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45745 } } peers { permanent_uuid: "2553fe98df544a7e85a4a5cbdfc779c9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46337 } } peers { permanent_uuid: "65c37adac3704fd9812201ccee5a9df3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34109 } } }
I20260928 05:16:25.089154 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:25.108403 23072 catalog_manager.cc:1259] Loaded cluster ID: 09fadf81f5b84bcda7ee2be72496ce30
I20260928 05:16:25.108587 23072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: loading cluster ID for follower catalog manager: success
I20260928 05:16:25.126650 23072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: acquiring CA information for follower catalog manager: success
I20260928 05:16:25.129431 23072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2553fe98df544a7e85a4a5cbdfc779c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:16:25.195483 22937 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.172781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.0.97.61:46337)
0928 05:16:25.172953 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.172958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.193239 (+ 20281us) server_negotiation.cc:407] Connection header received
0928 05:16:25.193782 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.193830 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.193926 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.194017 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.195234 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60384: BlockingRecv error: recv got EOF from 127.0.0.1:60384 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 05:16:25.196704 22862 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.170851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.0.97.62:45745)
0928 05:16:25.171083 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.171089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.187172 (+ 16083us) server_negotiation.cc:407] Connection header received
0928 05:16:25.195046 (+ 7874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.195053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.195142 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.195222 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.196446 (+ 1224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46130: BlockingRecv error: recv got EOF from 127.0.0.1:46130 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 05:16:25.198076 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.182775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.97.60:34109)
0928 05:16:25.182978 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.182984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.196294 (+ 13310us) server_negotiation.cc:407] Connection header received
0928 05:16:25.196582 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.196588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.196687 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.196765 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.197828 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52420: BlockingRecv error: recv got EOF from 127.0.0.1:52420 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 05:16:25.256204 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.234957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.97.61:46337)
0928 05:16:25.235167 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.235177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.247746 (+ 12569us) server_negotiation.cc:407] Connection header received
0928 05:16:25.248340 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.248350 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.248601 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.248752 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.249534 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.250704 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.252736 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.253440 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.253545 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.254506 (+ 961us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.255343 (+ 837us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.255479 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.255751 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.255851 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 05:16:25.305552 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.291979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.0.97.60:34109)
0928 05:16:25.292204 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.292215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.303497 (+ 11282us) server_negotiation.cc:407] Connection header received
0928 05:16:25.303777 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.303783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.303953 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.304082 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.305277 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52436: BlockingRecv error: recv got EOF from 127.0.0.1:52436 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 05:16:25.366418 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.348514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.97.61:46337)
0928 05:16:25.348706 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.348712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.359229 (+ 10517us) server_negotiation.cc:407] Connection header received
0928 05:16:25.359812 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.359820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.359921 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.360002 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.360751 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.361917 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.363987 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.364730 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.364854 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.365483 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.365675 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.365749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.366011 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.366104 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 05:16:25.409018 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.400127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.0.97.60:34109)
0928 05:16:25.400398 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.400408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.407163 (+ 6755us) server_negotiation.cc:407] Connection header received
0928 05:16:25.407421 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.407427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.407529 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.407652 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.408738 (+ 1086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52450: BlockingRecv error: recv got EOF from 127.0.0.1:52450 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 05:16:25.466387 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.447140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.97.61:46337)
0928 05:16:25.447360 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.447369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.459366 (+ 11997us) server_negotiation.cc:407] Connection header received
0928 05:16:25.459878 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.459885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.459999 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.460068 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.460777 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.461903 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.464095 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.464774 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.464892 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.465505 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.465656 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.465714 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.465953 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.466048 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 05:16:25.514740 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.503724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.0.97.60:34109)
0928 05:16:25.503900 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.503906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.513147 (+ 9241us) server_negotiation.cc:407] Connection header received
0928 05:16:25.513407 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.513412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.513504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.513620 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.514463 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52460: BlockingRecv error: recv got EOF from 127.0.0.1:52460 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:16:25.528152 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.522733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.97.61:46337)
0928 05:16:25.522978 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.522986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.523340 (+ 354us) server_negotiation.cc:407] Connection header received
0928 05:16:25.523558 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.523571 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.523638 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.523745 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.524153 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.524950 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.526328 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.526854 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.527004 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.527414 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.527519 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.527629 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.527761 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.527848 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260928 05:16:25.536190 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.534314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.0.97.60:34109)
0928 05:16:25.534663 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.534669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.534700 (+ 31us) server_negotiation.cc:407] Connection header received
0928 05:16:25.534916 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.534923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.535024 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.535100 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.536002 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52476: BlockingRecv error: recv got EOF from 127.0.0.1:52476 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 05:16:25.553488 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.547699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.0.97.61:46337)
0928 05:16:25.548094 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.548103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.548637 (+ 534us) server_negotiation.cc:407] Connection header received
0928 05:16:25.548858 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.548874 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.548945 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.549045 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.549486 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.550266 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.551676 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.552140 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.552282 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.552673 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.552804 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.552904 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.553049 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.553134 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
W20260928 05:16:25.561533 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.559373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.0.97.60:34109)
0928 05:16:25.559605 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.559613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.560138 (+ 525us) server_negotiation.cc:407] Connection header received
0928 05:16:25.560333 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.560339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.560423 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.560533 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.561290 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52490: BlockingRecv error: recv got EOF from 127.0.0.1:52490 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 05:16:25.587049 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.581062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.0.97.61:46337)
0928 05:16:25.581251 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.581258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.581585 (+ 327us) server_negotiation.cc:407] Connection header received
0928 05:16:25.581754 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.581761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.581874 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.582010 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.582458 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.583676 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.585039 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.585706 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.585852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.586241 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.586383 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.586492 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.586649 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.586752 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 05:16:25.594581 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.592706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.0.97.60:34109)
0928 05:16:25.592938 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.592945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.593204 (+ 259us) server_negotiation.cc:407] Connection header received
0928 05:16:25.593341 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.593346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.593450 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.593559 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.594316 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52498: BlockingRecv error: recv got EOF from 127.0.0.1:52498 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 05:16:25.638398 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.632076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.97.61:46337)
0928 05:16:25.632313 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.632320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.632633 (+ 313us) server_negotiation.cc:407] Connection header received
0928 05:16:25.632894 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.632898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.632980 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.633062 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.633573 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.634467 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.636450 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.636953 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.637063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.637612 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.637725 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.637796 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.637970 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.638040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 05:16:25.678241 23009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:25.674962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.0.97.60:34109)
0928 05:16:25.675139 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.675145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.676775 (+ 1630us) server_negotiation.cc:407] Connection header received
0928 05:16:25.677000 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.677006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.677105 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.677225 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:25.677981 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52504: BlockingRecv error: recv got EOF from 127.0.0.1:52504 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:16:25.750356 23023 catalog_manager.cc:1259] Loaded cluster ID: 09fadf81f5b84bcda7ee2be72496ce30
I20260928 05:16:25.750451 23023 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: loading cluster ID for follower catalog manager: success
I20260928 05:16:25.751480 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.746216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.97.61:46337)
0928 05:16:25.746395 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.746399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.746769 (+ 370us) server_negotiation.cc:407] Connection header received
0928 05:16:25.746945 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.746949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.747034 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.747131 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.747504 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.748221 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.749559 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.750035 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.750233 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.750684 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.750807 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.750861 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.751063 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.751144 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 05:16:25.753518 23023 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: acquiring CA information for follower catalog manager: success
I20260928 05:16:25.755728 23023 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65c37adac3704fd9812201ccee5a9df3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:16:25.762931 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.757444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.0.97.60:34109)
0928 05:16:25.757674 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.757680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.757875 (+ 195us) server_negotiation.cc:407] Connection header received
0928 05:16:25.758037 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.758041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.758156 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.758249 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.758672 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.759384 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.760799 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.761219 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.761315 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.761754 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.762264 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.762380 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.762479 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.762609 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 05:16:25.776783 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.769096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.97.62:45745)
0928 05:16:25.769657 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.769665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.770096 (+ 431us) server_negotiation.cc:407] Connection header received
0928 05:16:25.770363 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.770367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.770616 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.770784 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.771345 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.772319 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.774478 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.774961 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.775109 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.775691 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.776142 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.776258 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.776394 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.776484 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":342,"threads_started":1}
I20260928 05:16:25.777325 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.769309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.0.97.60:34109)
0928 05:16:25.769836 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.769841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.771894 (+ 2053us) server_negotiation.cc:407] Connection header received
0928 05:16:25.772108 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.772116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.772206 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.772276 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.773036 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.773727 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.775305 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.775851 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.775994 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.776469 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.776599 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.776668 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.776856 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.777074 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380}
I20260928 05:16:25.778627 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:25.769101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.0.97.61:46337)
0928 05:16:25.769312 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:25.769322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:25.771702 (+ 2380us) server_negotiation.cc:407] Connection header received
0928 05:16:25.771831 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:25.771837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:25.771921 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:25.772015 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:25.772577 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.773726 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.776454 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:25.777152 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:25.777250 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:25.777871 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:25.778013 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:25.778069 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:25.778256 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 05:16:25.778338 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:16:25.783190 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22791
I20260928 05:16:25.784415 22853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:25.893812 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22791
I20260928 05:16:25.915333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22865
I20260928 05:16:25.916442 22927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:26.023455 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22865
I20260928 05:16:26.044636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 22939
I20260928 05:16:26.045671 23001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:26.155973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 22939
2026-09-28T05:16:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4310 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 05:16:26 dist-test-slave-6718 krb5kdc[23176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 05:16:26 dist-test-slave-6718 krb5kdc[23176](info): set up 2 sockets
Sep 28 05:16:26 dist-test-slave-6718 krb5kdc[23176](info): commencing operation
krb5kdc: starting...
W20260928 05:16:27.279181 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/test-user.keytab.
Sep 28 05:16:27 dist-test-slave-6718 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T05:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T05:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 05:16:27.558350 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36671
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36671,127.0.97.61:40587,127.0.97.60:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 05:16:27.888860 23189 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:16:27.889253 23189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:27.889392 23189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:27.889532 23189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:27.896879 23189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:27.897004 23189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:27.897125 23189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:27.897209 23189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:27.897347 23189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:27.907202 23189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36671,127.0.97.61:40587,127.0.97.60:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:27.908532 23189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:27.909833 23189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:27.922005 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:27.922042 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:27.921978 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:27.923197 23189 server_base.cc:1034] running on GCE node
I20260928 05:16:27.923982 23189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:27.925395 23189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:27.926648 23189 hybrid_clock.cc:648] HybridClock initialized: now 1790572587926563 us; error 62 us; skew 500 ppm
I20260928 05:16:27.926985 23189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:27 dist-test-slave-6718 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:27.930326 23189 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 05:16:27.931902 23189 webserver.cc:460] Webserver started at http://127.0.97.62:44979/ using document root <none> and password file <none>
I20260928 05:16:27.932610 23189 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:27.932735 23189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:27.933049 23189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:27.936069 23189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data/instance:
uuid: "98d1adbddc454ad9bd8d7ecfc3ded406"
format_stamp: "Formatted at 2026-09-28 05:16:27 on dist-test-slave-6718"
server_key: "d9f6b04f448288b926cca3105658dec8"
server_key_iv: "00b8fe22ea2a826974f6660991863d80"
server_key_version: "encryptionkey@0"
I20260928 05:16:27.936841 23189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal/instance:
uuid: "98d1adbddc454ad9bd8d7ecfc3ded406"
format_stamp: "Formatted at 2026-09-28 05:16:27 on dist-test-slave-6718"
server_key: "d9f6b04f448288b926cca3105658dec8"
server_key_iv: "00b8fe22ea2a826974f6660991863d80"
server_key_version: "encryptionkey@0"
I20260928 05:16:27.942134 23189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 05:16:27.945624 23204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:27.947059 23189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 05:16:27.947269 23189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal
uuid: "98d1adbddc454ad9bd8d7ecfc3ded406"
format_stamp: "Formatted at 2026-09-28 05:16:27 on dist-test-slave-6718"
server_key: "d9f6b04f448288b926cca3105658dec8"
server_key_iv: "00b8fe22ea2a826974f6660991863d80"
server_key_version: "encryptionkey@0"
I20260928 05:16:27.947504 23189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:27.970139 23189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:27.975199 23189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:27.975528 23189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:27.995288 23189 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36671
I20260928 05:16:27.995299 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36671 every 8 connection(s)
I20260928 05:16:27.996762 23189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/data/info.pb
I20260928 05:16:28.000541 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:28.003252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 23189
I20260928 05:16:28.003473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-0/wal/instance
I20260928 05:16:28.003913 388 external_mini_cluster.cc:1428] Setting key f3dc9a656ea8a2930ce6893a7c72f4e2
I20260928 05:16:28.007004 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:28 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260928 05:16:28.032233 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.010170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.97.62:36671)
0928 05:16:28.011070 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:28.011090 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:28.011493 (+ 403us) server_negotiation.cc:407] Connection header received
0928 05:16:28.014136 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:28.014142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:28.014390 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:28.014669 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:28.016291 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.017520 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.019815 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.020388 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.020530 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.024740 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:28.024749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:28.024763 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:28.024792 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:28.027766 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.028466 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.028472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.028475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.028609 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.029017 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.029028 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.029031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.029587 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:28.029793 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:28.030312 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 05:16:28.030699 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"spinlock_wait_cycles":20864,"thread_start_us":433,"threads_started":1}
I20260928 05:16:28.034619 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.010171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36671 (local address 127.0.0.1:33220)
0928 05:16:28.011339 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.011372 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:28.013732 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:28.014675 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:28.014685 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:28.015517 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:28.016045 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.016073 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.017684 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.017693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.019643 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.019661 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.020635 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.020649 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.020901 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.021627 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:28.021663 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:28.024485 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:28.027912 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.027925 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.027938 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.028325 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.028729 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.028733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.028737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.028906 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.029729 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:28.029738 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:28.029947 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 05:16:28.030186 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 05:16:28.030811 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":153,"thread_start_us":774,"threads_started":1}
I20260928 05:16:28.037719 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:16:28.039021 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.038611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46148)
0928 05:16:28.038776 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.038907 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:16:28.039309 23206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
I20260928 05:16:28.040902 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.040527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46150)
0928 05:16:28.040664 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.040736 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 05:16:28.041131 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 05:16:28.062711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40587
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36671,127.0.97.61:40587,127.0.97.60:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:16:28.090514 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } attempt: 1
I20260928 05:16:28.091815 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.091288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46152)
0928 05:16:28.091516 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.091636 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 05:16:28.092949 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.092562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46168)
0928 05:16:28.092711 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.092801 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:16:28.093171 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
I20260928 05:16:28.161418 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } attempt: 2
I20260928 05:16:28.162741 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.162096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46182)
0928 05:16:28.162374 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.162563 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 05:16:28.163929 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.163429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46184)
0928 05:16:28.163625 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.163742 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 05:16:28.164143 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
I20260928 05:16:28.255442 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } attempt: 3
I20260928 05:16:28.256780 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.256172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46196)
0928 05:16:28.256440 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.256575 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 05:16:28.257954 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.257466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46212)
0928 05:16:28.257652 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.257760 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 05:16:28.258177 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
W20260928 05:16:28.393149 23263 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:16:28.393476 23263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:28.393545 23263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:28.393626 23263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:28.400931 23263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:28.401026 23263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:28.401082 23263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:28.401134 23263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:28.401191 23263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:28.401445 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } attempt: 4
I20260928 05:16:28.402698 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.402068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46226)
0928 05:16:28.402385 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.402501 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 05:16:28.403806 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.403421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46228)
0928 05:16:28.403586 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.403665 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 05:16:28.404021 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40587: connect: Connection refused (error 111)
I20260928 05:16:28.411772 23263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36671,127.0.97.61:40587,127.0.97.60:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:28.413010 23263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:28.414214 23263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:28.426184 23268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:28.426219 23271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:28.426281 23269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:28.432343 23263 server_base.cc:1034] running on GCE node
I20260928 05:16:28.433336 23263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:28.434916 23263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:28.436189 23263 hybrid_clock.cc:648] HybridClock initialized: now 1790572588436112 us; error 34 us; skew 500 ppm
I20260928 05:16:28.436570 23263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:28 dist-test-slave-6718 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:28.440048 23263 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 05:16:28.441642 23263 webserver.cc:460] Webserver started at http://127.0.97.61:37595/ using document root <none> and password file <none>
I20260928 05:16:28.442384 23263 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:28.442517 23263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:28.442902 23263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:28.445861 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data/instance:
uuid: "3d316ec0440743c593804e078cd1bbc0"
format_stamp: "Formatted at 2026-09-28 05:16:28 on dist-test-slave-6718"
server_key: "720619172299452d1287b8859a04efc5"
server_key_iv: "f26a8be46123fe9918e27200094382e1"
server_key_version: "encryptionkey@0"
I20260928 05:16:28.446740 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal/instance:
uuid: "3d316ec0440743c593804e078cd1bbc0"
format_stamp: "Formatted at 2026-09-28 05:16:28 on dist-test-slave-6718"
server_key: "720619172299452d1287b8859a04efc5"
server_key_iv: "f26a8be46123fe9918e27200094382e1"
server_key_version: "encryptionkey@0"
I20260928 05:16:28.451809 23263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 05:16:28.455232 23278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:28.456511 23263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 05:16:28.456692 23263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal
uuid: "3d316ec0440743c593804e078cd1bbc0"
format_stamp: "Formatted at 2026-09-28 05:16:28 on dist-test-slave-6718"
server_key: "720619172299452d1287b8859a04efc5"
server_key_iv: "f26a8be46123fe9918e27200094382e1"
server_key_version: "encryptionkey@0"
I20260928 05:16:28.456909 23263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:28.473573 23263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:28.477424 23263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:28.477741 23263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:28.496627 23263 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40587
I20260928 05:16:28.496660 23329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40587 every 8 connection(s)
I20260928 05:16:28.498081 23263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/data/info.pb
I20260928 05:16:28.501904 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:28.507617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 23263
I20260928 05:16:28.507807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-1/wal/instance
I20260928 05:16:28.508380 388 external_mini_cluster.cc:1428] Setting key 582c333d08b36f0738ad92afb02ec5ef
I20260928 05:16:28.510471 23330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:28 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:16:28.535030 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.513818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.0.97.62:36671)
0928 05:16:28.514061 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:28.514069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:28.515583 (+ 1514us) server_negotiation.cc:407] Connection header received
0928 05:16:28.518859 (+ 3276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:28.518868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:28.518987 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:28.519112 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:28.520587 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.521760 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.523358 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.524097 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.524241 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.527700 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:28.527709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:28.527717 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:28.527762 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:28.531566 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.532382 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.532389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.532403 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.532568 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.533018 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.533025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.533032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.533477 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:28.533647 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:28.534357 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 05:16:28.534651 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 05:16:28.535854 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.514303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36671 (local address 127.0.0.1:33228)
0928 05:16:28.515394 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.515444 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:28.518227 (+ 2783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:28.519315 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:28.519323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:28.519853 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:28.520411 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.520428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.521912 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.521916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.523212 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.523224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.524315 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.524319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.524519 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.525232 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:28.525255 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:28.527466 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:28.531746 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.531757 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.531776 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.532243 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.532690 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.532693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.532696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.532887 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.533613 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:28.533619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:28.533950 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 05:16:28.534187 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 05:16:28.534646 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":415,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260928 05:16:28.537603 23330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 28 05:16:28 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:28.552402 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.538452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46232)
0928 05:16:28.538732 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.538753 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:28.538856 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:28.539758 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:28.539762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:28.540203 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:28.540327 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.540335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.541727 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.541732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.542613 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.542624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.543463 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.543469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.543638 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.544208 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:28.544225 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:28.546326 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:28.549500 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.549508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.549515 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.550059 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.550407 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.550414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.550439 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.550640 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.551252 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:28.551260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:28.551443 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 05:16:28.551591 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 05:16:28.551890 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 05:16:28.552435 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.538664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.0.97.61:40587)
0928 05:16:28.539298 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:28.539304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:28.539326 (+ 22us) server_negotiation.cc:407] Connection header received
0928 05:16:28.539368 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:28.539374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:28.539617 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:28.539776 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:28.540506 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.541581 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.542739 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.543295 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.543439 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.546569 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:28.546585 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:28.546598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:28.546627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:28.549371 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.550189 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.550195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.550197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.550299 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.550747 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.550750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.550752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.551110 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:28.551279 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:28.551781 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 05:16:28.552042 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":379,"threads_started":1}
I20260928 05:16:28.555348 23330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:16:28.556520 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.556044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37568)
0928 05:16:28.556244 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.556367 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 05:16:28.556825 23281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.558362 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.558030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37570)
0928 05:16:28.558183 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.558257 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 05:16:28.558601 23330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 05:16:28.569870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
/tmp/dist-test-taskzfOJKc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39447
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36671,127.0.97.61:40587,127.0.97.60:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 05:16:28.607985 23330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 1
I20260928 05:16:28.609306 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.608839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37572)
0928 05:16:28.609063 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.609154 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 05:16:28.610296 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.609956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37584)
0928 05:16:28.610100 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.610192 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 05:16:28.610487 23330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.678756 23330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 2
I20260928 05:16:28.679913 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.679465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37600)
0928 05:16:28.679650 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.679759 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 05:16:28.681044 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.680650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37606)
0928 05:16:28.680826 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.680934 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 05:16:28.681231 23330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.703295 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } attempt: 5
Sep 28 05:16:28 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:28.718089 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.704087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46234)
0928 05:16:28.704314 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.704336 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:28.704481 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:28.704896 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:28.704901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:28.705417 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:28.705645 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.705657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.706752 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.706757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.708262 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:28.708281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.709141 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.709147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:28.709369 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.710024 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:28.710048 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:28.712595 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:28.715705 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.715710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.715714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.716101 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.716496 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:28.716501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:28.716504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:28.716706 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:28.717370 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:28.717374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:28.717509 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 05:16:28.717623 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 05:16:28.717832 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 05:16:28.718338 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.704191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.0.97.61:40587)
0928 05:16:28.704377 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:28.704381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:28.704401 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:16:28.704625 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:28.704629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:28.704710 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:28.704795 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:28.705805 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.706596 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.708445 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:28.708982 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:28.709084 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:28.712815 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:28.712820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:28.712823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:28.712861 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:28.715550 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.716228 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.716236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.716239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.716372 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:28.716857 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:28.716864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:28.716867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:28.717243 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:28.717400 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:28.717754 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 05:16:28.718039 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 05:16:28.719429 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } has no permanent_uuid. Determining permanent_uuid...
I20260928 05:16:28.720559 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.720172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37608)
0928 05:16:28.720343 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.720427 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 05:16:28.721503 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.721186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37610)
0928 05:16:28.721330 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.721402 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:16:28.721724 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.772468 23330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 3
I20260928 05:16:28.772961 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 1
I20260928 05:16:28.773767 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.773210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37620)
0928 05:16:28.773456 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.773560 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 05:16:28.774048 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.773544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37632)
0928 05:16:28.773761 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.773872 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 05:16:28.775080 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.774744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37650)
0928 05:16:28.774914 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.774980 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 05:16:28.775101 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.774600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37646)
0928 05:16:28.774871 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.774966 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260928 05:16:28.775393 23330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
W20260928 05:16:28.775446 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.843683 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 2
I20260928 05:16:28.844882 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.844400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37652)
0928 05:16:28.844598 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.844710 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 05:16:28.845906 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.845562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37660)
0928 05:16:28.845719 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.845800 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 05:16:28.846130 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
W20260928 05:16:28.903492 23337 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 05:16:28.903855 23337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 05:16:28.903986 23337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 05:16:28.904132 23337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 05:16:28.911375 23337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 05:16:28.911473 23337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 05:16:28.911564 23337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 05:16:28.911640 23337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 05:16:28.911728 23337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 05:16:28.918678 23330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 4
I20260928 05:16:28.919889 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.919368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37662)
0928 05:16:28.919595 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.919689 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 05:16:28.920971 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.920570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37678)
0928 05:16:28.920794 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.920866 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 05:16:28.921233 23330 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.922212 23337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36671,127.0.97.61:40587,127.0.97.60:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 04:16:20 UTC on ccycloud.quasar-qsxtlg.root.comops.site
build id 4676562
ASAN enabled
I20260928 05:16:28.923537 23337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 05:16:28.924846 23337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 05:16:28.936969 23345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:28.937034 23342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 05:16:28.937011 23343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 05:16:28.937565 23337 server_base.cc:1034] running on GCE node
I20260928 05:16:28.938354 23337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 05:16:28.939898 23337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 05:16:28.941165 23337 hybrid_clock.cc:648] HybridClock initialized: now 1790572588941064 us; error 55 us; skew 500 ppm
I20260928 05:16:28.941494 23337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 05:16:28 dist-test-slave-6718 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 05:16:28.944828 23337 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 05:16:28.946306 23337 webserver.cc:460] Webserver started at http://127.0.97.60:45183/ using document root <none> and password file <none>
I20260928 05:16:28.947072 23337 fs_manager.cc:359] Metadata directory not provided
I20260928 05:16:28.947163 23337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 05:16:28.947507 23337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 05:16:28.950469 23337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data/instance:
uuid: "79098c2857014352831812072e70830c"
format_stamp: "Formatted at 2026-09-28 05:16:28 on dist-test-slave-6718"
server_key: "c114476e324157090a5cbab19f01bc2c"
server_key_iv: "16c2d3945fd42b063911b3b83cc526d0"
server_key_version: "encryptionkey@0"
I20260928 05:16:28.951340 23337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal/instance:
uuid: "79098c2857014352831812072e70830c"
format_stamp: "Formatted at 2026-09-28 05:16:28 on dist-test-slave-6718"
server_key: "c114476e324157090a5cbab19f01bc2c"
server_key_iv: "16c2d3945fd42b063911b3b83cc526d0"
server_key_version: "encryptionkey@0"
I20260928 05:16:28.952339 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 3
I20260928 05:16:28.953410 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.952927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37686)
0928 05:16:28.953174 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.953257 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 05:16:28.954525 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:28.954108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37698)
0928 05:16:28.954314 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:28.954381 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 05:16:28.954912 23256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39447: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39447: connect: Connection refused (error 111)
I20260928 05:16:28.957185 23337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 05:16:28.960829 23352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 05:16:28.962265 23337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 05:16:28.962440 23337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal
uuid: "79098c2857014352831812072e70830c"
format_stamp: "Formatted at 2026-09-28 05:16:28 on dist-test-slave-6718"
server_key: "c114476e324157090a5cbab19f01bc2c"
server_key_iv: "16c2d3945fd42b063911b3b83cc526d0"
server_key_version: "encryptionkey@0"
I20260928 05:16:28.962746 23337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 05:16:28.985385 23337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 05:16:28.989976 23337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 05:16:28.990255 23337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 05:16:29.008945 23337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39447
I20260928 05:16:29.008920 23403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39447 every 8 connection(s)
I20260928 05:16:29.010248 23337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/data/info.pb
I20260928 05:16:29.014166 23404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 05:16:29.014490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu as pid 23337
I20260928 05:16:29.014694 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzfOJKc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790572228671169-388-0/minicluster-data/master-2/wal/instance
I20260928 05:16:29.015229 388 external_mini_cluster.cc:1428] Setting key eb3e6d44186b7d232076909bb52b9606
I20260928 05:16:29.023658 23404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 05:16:29.038133 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.017043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.0.97.60:39447)
0928 05:16:29.017812 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.017824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.017866 (+ 42us) server_negotiation.cc:407] Connection header received
0928 05:16:29.019524 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.019546 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.019994 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.020417 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.021932 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.023284 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.024633 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.025313 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.025540 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.030236 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.030244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.030256 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.030292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.033754 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.034597 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.034612 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.034622 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.034832 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.035297 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.035301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.035303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.035805 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.036147 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.036519 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.036991 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":350,"threads_started":1}
I20260928 05:16:29.041468 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.026174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.0.97.62:36671)
0928 05:16:29.026448 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.026456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.027117 (+ 661us) server_negotiation.cc:407] Connection header received
0928 05:16:29.027293 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.027299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.027420 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.027538 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.028730 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.029498 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.031143 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.031610 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.031764 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.035300 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.035307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.035313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.035358 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.038505 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.039242 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.039248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.039251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.039420 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.039786 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.039793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.039798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.040127 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.040291 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.040873 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.041096 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 05:16:29.041496 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.026130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36671 (local address 127.0.0.1:33242)
0928 05:16:29.026952 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:29.026999 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:29.027159 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:29.027592 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:29.027597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:29.028247 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:29.028574 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.028586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.029660 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.029666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.030968 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.030980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.031853 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.031862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.032082 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.032656 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:29.032674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:29.035076 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:29.038691 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.038703 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.038710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.039111 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.039538 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.039541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.039543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.039666 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.040266 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:29.040271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:29.040497 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 05:16:29.040715 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 05:16:29.041045 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":412,"threads_started":1}
I20260928 05:16:29.043730 23404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:29.057165 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.041462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.97.62:36671)
0928 05:16:29.041800 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.041808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.041829 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:16:29.041886 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.041894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.041988 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.042095 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.043603 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.044513 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.046114 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.046508 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.046688 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.051071 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.051077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.051082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.051143 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.054637 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.055266 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.055274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.055279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.055417 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.055795 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.055801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.055805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.056077 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.056347 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.056618 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.056820 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":495,"threads_started":1}
I20260928 05:16:29.057637 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.044528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40587 (local address 127.0.0.1:46244)
0928 05:16:29.044680 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:29.044700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:29.044796 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:29.045140 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:29.045143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:29.045467 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:29.045600 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.045608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.046793 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.046798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.047820 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.047831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.048830 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.048834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.048961 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.049473 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:29.049489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:29.051709 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:29.055337 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.055342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.055346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.055693 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.056077 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.056084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.056087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.056224 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.056722 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:29.056726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:29.056882 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 05:16:29.057028 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 05:16:29.057300 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 05:16:29.057813 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.044544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.0.97.61:40587)
0928 05:16:29.044733 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.044739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.044759 (+ 20us) server_negotiation.cc:407] Connection header received
0928 05:16:29.044908 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.044914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.044991 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.045094 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.045741 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.046622 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.047973 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.048683 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.048826 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.051915 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.051922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.051926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.051994 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.055189 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.055817 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.055824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.055828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.055966 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.056332 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.056345 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.056348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.056600 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.056760 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.057262 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.057478 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 05:16:29.058775 23404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 05:16:29.072255 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.059492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37704)
0928 05:16:29.059650 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:29.059668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:29.059759 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:29.060139 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:29.060144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:29.060474 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:29.060658 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.060672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.061946 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.061951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.063032 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.063043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.063953 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.063959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.064164 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.064714 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:29.064735 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:29.066740 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:29.070036 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.070039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.070041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.070379 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.070811 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.070817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.070823 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.070964 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.071561 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:29.071564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:29.071677 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 05:16:29.071782 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 05:16:29.071982 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 05:16:29.072700 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.059716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.0.97.60:39447)
0928 05:16:29.059852 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.059856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.059869 (+ 13us) server_negotiation.cc:407] Connection header received
0928 05:16:29.059903 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.059906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.059962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.060075 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.060804 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.061820 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.063160 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.063765 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.063890 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.066894 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.066900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.066905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.066941 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.069912 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.070500 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.070507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.070512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.070682 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.071074 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.071079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.071083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.071447 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.071594 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.071913 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.072169 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 05:16:29.075328 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.059002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.97.61:40587)
0928 05:16:29.059144 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.059149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.059791 (+ 642us) server_negotiation.cc:407] Connection header received
0928 05:16:29.059845 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.059852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.059919 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.059992 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.061434 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.062403 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.063865 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.064488 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.064621 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.069137 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.069143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.069148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.069186 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.072380 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.073131 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.073136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.073139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.073311 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.073779 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.073782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.073785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.074190 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.074457 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.074790 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.075024 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 05:16:29.078507 23404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: Bootstrap starting.
I20260928 05:16:29.080045 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 05:16:29.082361 23404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:29.083420 23404 log.cc:824] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:29.083062 23231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33256
I20260928 05:16:29.086086 23404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: No bootstrap required, opened a new log
I20260928 05:16:29.092885 23404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.093199 23404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79098c2857014352831812072e70830c, State: Initialized, Role: FOLLOWER
I20260928 05:16:29.093621 23404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.094502 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.094933 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.095667 23404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: configured and running, proceeding with master startup.
W20260928 05:16:29.104803 23422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 05:16:29.104979 23422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 05:16:29.102730 23337 master_runner.cc:186] Error getting master registration for 127.0.97.62:36671: OK, instance_id { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" instance_seqno: 1790572587989600 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:16:29.113930 23337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:16:29.132157 23256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 4
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:16:29.148543 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.133090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.0.97.60:39447)
0928 05:16:29.133406 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.133415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.133442 (+ 27us) server_negotiation.cc:407] Connection header received
0928 05:16:29.133517 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.133521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.133586 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.133703 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.134573 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.135395 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.136751 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.137398 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.137548 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.141754 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.141759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.141764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.141795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.145581 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.146295 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.146301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.146305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.146448 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.146932 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.146940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.146944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.147289 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.147475 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.147903 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.148118 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 05:16:29.148543 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.132970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37706)
0928 05:16:29.133188 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:29.133214 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:29.133362 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:29.133750 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:29.133756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:29.134200 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:29.134382 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.134392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.135559 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.135568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.136588 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.136601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.137569 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.137576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.137792 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.138462 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:29.138490 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:29.141546 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:29.145744 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.145749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.145754 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.146161 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.146625 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.146633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.146637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.146812 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.147450 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:29.147455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:29.147619 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 05:16:29.147771 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 05:16:29.148103 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 05:16:29.154718 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406: Bootstrap starting.
I20260928 05:16:29.158849 23256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:29.160210 23256 log.cc:824] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:29.163786 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406: No bootstrap required, opened a new log
I20260928 05:16:29.171438 23256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.171849 23256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d1adbddc454ad9bd8d7ecfc3ded406, State: Initialized, Role: FOLLOWER
I20260928 05:16:29.172346 23256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.173372 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.173836 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.174666 23256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:29.174779 23231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 05:16:29.175027 23231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:29.177779 23231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.179252 23231 leader_election.cc:288] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [CANDIDATE]: Term 1 election: Requested vote from peers 3d316ec0440743c593804e078cd1bbc0 (127.0.97.61:40587), 79098c2857014352831812072e70830c (127.0.97.60:39447)
I20260928 05:16:29.180380 23379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79098c2857014352831812072e70830c"
I20260928 05:16:29.180895 23379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:29.184695 23379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d1adbddc454ad9bd8d7ecfc3ded406 in term 1.
I20260928 05:16:29.185448 23207 leader_election.cc:302] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79098c2857014352831812072e70830c, 98d1adbddc454ad9bd8d7ecfc3ded406; no voters:
I20260928 05:16:29.186053 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 05:16:29.198827 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [term 1 LEADER]: Becoming Leader. State: Replica: 98d1adbddc454ad9bd8d7ecfc3ded406, State: Running, Role: LEADER
I20260928 05:16:29.199776 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.207722 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98d1adbddc454ad9bd8d7ecfc3ded406. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.208398 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:29.218773 23436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 05:16:29.220451 23330 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } attempt: 5
W20260928 05:16:29.228480 23210 tablet.cc:2369] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 05:16:29 dist-test-slave-6718 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790572588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 05:16:29.238926 23436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 05:16:29.242436 23358 tablet.cc:2369] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 05:16:29.263149 23379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [term 1 FOLLOWER]: Refusing update from remote peer 98d1adbddc454ad9bd8d7ecfc3ded406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:16:29.264700 23429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:16:29.278172 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.221220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39447 (local address 127.0.0.1:37708)
0928 05:16:29.221714 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 05:16:29.221739 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 05:16:29.221913 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 05:16:29.222904 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 05:16:29.222909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 05:16:29.223560 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 05:16:29.223860 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.223878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.226074 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.226083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.228268 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 05:16:29.228285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.229795 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.229805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 05:16:29.230489 (+ 684us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.231548 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 05:16:29.231578 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 05:16:29.237234 (+ 5656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 05:16:29.245788 (+ 8554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.245796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.245801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.257041 (+ 11240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.272014 (+ 14973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 05:16:29.272021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 05:16:29.272026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 05:16:29.272280 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 05:16:29.276960 (+ 4680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 05:16:29.276966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 05:16:29.277171 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 05:16:29.277347 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 05:16:29.277730 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260928 05:16:29.279484 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.222184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.0.97.60:39447)
0928 05:16:29.222371 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.222377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.222398 (+ 21us) server_negotiation.cc:407] Connection header received
0928 05:16:29.222465 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.222470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.222586 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.222696 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.224106 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.225804 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.228519 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.229248 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.229394 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.237817 (+ 8423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.237825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.237830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.237877 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.245237 (+ 7360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.257352 (+ 12115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.257397 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.257404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.257770 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.272515 (+ 14745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.272522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.272526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.272949 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.273128 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.278775 (+ 5647us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.279056 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 05:16:29.285723 23330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: Bootstrap starting.
I20260928 05:16:29.290235 23330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: Neither blocks nor log segments found. Creating new log.
I20260928 05:16:29.292941 23451 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 05:16:29.296018 23330 log.cc:824] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: Log is configured to *not* fsync() on all Append() calls
I20260928 05:16:29.297801 23189 master_runner.cc:186] Error getting master registration for 127.0.97.61:40587: OK, instance_id { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" instance_seqno: 1790572588491047 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 05:16:29.310595 23189 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 05:16:29.325412 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98d1adbddc454ad9bd8d7ecfc3ded406. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.325801 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.341032 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.341370 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:29.342459 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.342733 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.345101 23436 catalog_manager.cc:1347] Generated new cluster ID: ea62b96cb6ef446a9afc656ddc486186
I20260928 05:16:29.345204 23436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 05:16:29.348757 23330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: No bootstrap required, opened a new log
I20260928 05:16:29.371733 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.248776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.0.97.61:40587)
0928 05:16:29.249036 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.249045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.304915 (+ 55870us) server_negotiation.cc:407] Connection header received
0928 05:16:29.305204 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.305212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.305309 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.305406 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.307741 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.308984 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.326391 (+ 17407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.327110 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.327225 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.355627 (+ 28402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.355635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.355641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.355694 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.359293 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.365574 (+ 6281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.365583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.365588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.365774 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.367594 (+ 1820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.367600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.367604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.367979 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.368254 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.371064 (+ 2810us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.371352 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:16:29.374634 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.217874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.97.62:36671)
0928 05:16:29.218130 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.218140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.300246 (+ 82106us) server_negotiation.cc:407] Connection header received
0928 05:16:29.300640 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.300646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.300744 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.300854 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.317441 (+ 16587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.318723 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.331946 (+ 13223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.332736 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.332858 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.347719 (+ 14861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.347726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.347731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.347774 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.351892 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.366262 (+ 14370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.366269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.366274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.366505 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.368728 (+ 2223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.368735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.368739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.369163 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.369417 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.373932 (+ 4515us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.374213 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 05:16:29.377681 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:29.258948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.0.97.60:39447)
0928 05:16:29.290662 (+ 31714us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.290670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.310978 (+ 20308us) server_negotiation.cc:407] Connection header received
0928 05:16:29.311094 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.311102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.311205 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.311326 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.313663 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.314939 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.329683 (+ 14744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:29.330845 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:29.331044 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:29.360862 (+ 29818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 05:16:29.360870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 05:16:29.360876 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 05:16:29.360922 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 05:16:29.364698 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.366992 (+ 2294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.366999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.367003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.367150 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 05:16:29.369857 (+ 2707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 05:16:29.369863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 05:16:29.369866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 05:16:29.370228 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 05:16:29.370482 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:29.376916 (+ 6434us) server_negotiation.cc:299] Negotiation successful
0928 05:16:29.377208 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21013,"thread_start_us":50476,"threads_started":1}
I20260928 05:16:29.387890 23436 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 05:16:29.390049 23436 catalog_manager.cc:1504] Loading token signing keys...
I20260928 05:16:29.389909 23330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.390343 23330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d316ec0440743c593804e078cd1bbc0, State: Initialized, Role: FOLLOWER
I20260928 05:16:29.390915 23330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } }
I20260928 05:16:29.409593 23330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 05:16:29.410112 23304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 05:16:29.412025 23436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406: Generated new TSK 0
I20260928 05:16:29.413306 23436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 05:16:29.414637 23305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d316ec0440743c593804e078cd1bbc0"
I20260928 05:16:29.414450 23304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [term 1 FOLLOWER]: Refusing update from remote peer 98d1adbddc454ad9bd8d7ecfc3ded406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 05:16:29.416293 23425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 05:16:29.419155 23305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d1adbddc454ad9bd8d7ecfc3ded406 in term 1.
I20260928 05:16:29.434696 23455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.435214 23455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.476789 23455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98d1adbddc454ad9bd8d7ecfc3ded406. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.477218 23455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.477626 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.478077 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 05:16:29.489250 23473 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 05:16:29.515017 23259 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.494912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.97.62:36671)
0928 05:16:29.495117 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.495123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.511402 (+ 16279us) server_negotiation.cc:407] Connection header received
0928 05:16:29.512834 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.512842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.512943 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.513023 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.514714 (+ 1691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33280: BlockingRecv error: recv got EOF from 127.0.0.1:33280 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 05:16:29.517614 23407 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.496628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.0.97.60:39447)
0928 05:16:29.496807 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.496814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.515796 (+ 18982us) server_negotiation.cc:407] Connection header received
0928 05:16:29.515962 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.515969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.516059 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.516161 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.517377 (+ 1216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37722: BlockingRecv error: recv got EOF from 127.0.0.1:37722 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:16:29.519236 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.500863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.0.97.61:40587)
0928 05:16:29.501062 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.501070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.511842 (+ 10772us) server_negotiation.cc:407] Connection header received
0928 05:16:29.513258 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.513264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.513389 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.513465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.519013 (+ 5548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46280: BlockingRecv error: recv got EOF from 127.0.0.1:46280 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:16:29.543732 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.536283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.0.97.61:40587)
0928 05:16:29.536477 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.536484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.541937 (+ 5453us) server_negotiation.cc:407] Connection header received
0928 05:16:29.542188 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.542194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.542338 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.542465 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.543451 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46282: BlockingRecv error: recv got EOF from 127.0.0.1:46282 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 05:16:29.551992 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.547921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.0.97.61:40587)
0928 05:16:29.548092 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.548099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.550421 (+ 2322us) server_negotiation.cc:407] Connection header received
0928 05:16:29.550666 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.550674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.550769 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.550868 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.551776 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46294: BlockingRecv error: recv got EOF from 127.0.0.1:46294 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 05:16:29.564581 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.560424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.97.61:40587)
0928 05:16:29.560620 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.560625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.563081 (+ 2456us) server_negotiation.cc:407] Connection header received
0928 05:16:29.563284 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.563291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.563399 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.563514 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.564327 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46300: BlockingRecv error: recv got EOF from 127.0.0.1:46300 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 05:16:29.577361 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.574924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.0.97.61:40587)
0928 05:16:29.575113 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.575120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.575726 (+ 606us) server_negotiation.cc:407] Connection header received
0928 05:16:29.575992 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.575998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.576100 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.576252 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.577102 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46302: BlockingRecv error: recv got EOF from 127.0.0.1:46302 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 05:16:29.591120 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.588868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.0.97.61:40587)
0928 05:16:29.589050 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.589057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.589573 (+ 516us) server_negotiation.cc:407] Connection header received
0928 05:16:29.589787 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.589796 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.589904 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.590031 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.590881 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46314: BlockingRecv error: recv got EOF from 127.0.0.1:46314 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 05:16:29.613242 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.610941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.0.97.61:40587)
0928 05:16:29.611131 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.611137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.611648 (+ 511us) server_negotiation.cc:407] Connection header received
0928 05:16:29.611872 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.611879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.611987 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.612107 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.612980 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46324: BlockingRecv error: recv got EOF from 127.0.0.1:46324 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 05:16:29.651772 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.648976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.97.61:40587)
0928 05:16:29.649178 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.649187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.649729 (+ 542us) server_negotiation.cc:407] Connection header received
0928 05:16:29.650031 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.650038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.650153 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.650227 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.651438 (+ 1211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 05:16:29.722131 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.719575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.0.97.61:40587)
0928 05:16:29.719811 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.719817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.720595 (+ 778us) server_negotiation.cc:407] Connection header received
0928 05:16:29.720823 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.720828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.720931 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.721022 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.721914 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46338: BlockingRecv error: recv got EOF from 127.0.0.1:46338 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 05:16:29.855598 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:29.853803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.0.97.61:40587)
0928 05:16:29.853978 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:29.853983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:29.854307 (+ 324us) server_negotiation.cc:407] Connection header received
0928 05:16:29.854478 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:29.854482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:29.854577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:29.854674 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:29.855321 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46348: BlockingRecv error: recv got EOF from 127.0.0.1:46348 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 05:16:29.867830 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d1adbddc454ad9bd8d7ecfc3ded406" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36671 } } peers { permanent_uuid: "3d316ec0440743c593804e078cd1bbc0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40587 } } peers { permanent_uuid: "79098c2857014352831812072e70830c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39447 } } }
I20260928 05:16:29.868196 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d1adbddc454ad9bd8d7ecfc3ded406 [sys.catalog]: This master's current role is: LEADER
I20260928 05:16:30.107167 23422 catalog_manager.cc:1259] Loaded cluster ID: ea62b96cb6ef446a9afc656ddc486186
I20260928 05:16:30.107266 23422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: loading cluster ID for follower catalog manager: success
I20260928 05:16:30.111119 23422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: acquiring CA information for follower catalog manager: success
I20260928 05:16:30.113771 23422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79098c2857014352831812072e70830c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 05:16:30.118052 23334 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 05:16:30.116518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.97.61:40587)
0928 05:16:30.116740 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:30.116746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:30.116969 (+ 223us) server_negotiation.cc:407] Connection header received
0928 05:16:30.117115 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:30.117119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:30.117191 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:30.117282 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 05:16:30.117890 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46364: BlockingRecv error: recv got EOF from 127.0.0.1:46364 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 05:16:30.493571 23473 catalog_manager.cc:1259] Loaded cluster ID: ea62b96cb6ef446a9afc656ddc486186
I20260928 05:16:30.493685 23473 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: loading cluster ID for follower catalog manager: success
I20260928 05:16:30.496534 23473 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: acquiring CA information for follower catalog manager: success
I20260928 05:16:30.498592 23473 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d316ec0440743c593804e078cd1bbc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 05:16:30.648727 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:30.641771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.97.61:40587)
0928 05:16:30.642642 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:30.642646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:30.642664 (+ 18us) server_negotiation.cc:407] Connection header received
0928 05:16:30.642730 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:30.642733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:30.642884 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:30.642998 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:30.643487 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.644423 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.646326 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.646808 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.646899 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:30.647488 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:30.648077 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:30.648197 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:30.648320 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 05:16:30.648391 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":521,"threads_started":1}
I20260928 05:16:30.661325 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:30.655167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.0.97.60:39447)
0928 05:16:30.656057 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:30.656065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:30.656084 (+ 19us) server_negotiation.cc:407] Connection header received
0928 05:16:30.656182 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:30.656185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:30.656324 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:30.656494 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:30.656882 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.657672 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.659010 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.659386 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.659513 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:30.660013 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:30.660635 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:30.660787 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:30.660973 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 05:16:30.661056 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":670,"threads_started":1}
I20260928 05:16:30.686761 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:30.667410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.0.97.62:36671)
0928 05:16:30.679152 (+ 11742us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:30.679161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:30.679190 (+ 29us) server_negotiation.cc:407] Connection header received
0928 05:16:30.679346 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:30.679355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:30.679625 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:30.679851 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:30.680576 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.681693 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.683889 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.684521 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.685168 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:30.685350 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:30.685801 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:30.686305 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:30.686359 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 05:16:30.686440 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11558,"thread_start_us":12562,"threads_started":1}
I20260928 05:16:30.687584 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:30.667715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.0.97.60:39447)
0928 05:16:30.673238 (+ 5523us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:30.673247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:30.679898 (+ 6651us) server_negotiation.cc:407] Connection header received
0928 05:16:30.680801 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:30.680808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:30.680933 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:30.680995 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:30.681878 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.683007 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.685433 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.686079 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.686174 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:30.686841 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:30.686980 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:30.687044 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:30.687258 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 05:16:30.687336 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5361}
I20260928 05:16:30.688508 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 05:16:30.667139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.97.61:40587)
0928 05:16:30.667331 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 05:16:30.667337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 05:16:30.680712 (+ 13375us) server_negotiation.cc:407] Connection header received
0928 05:16:30.680985 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 05:16:30.680991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 05:16:30.681079 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 05:16:30.681140 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 05:16:30.681968 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.683130 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.685441 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 05:16:30.686079 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 05:16:30.686174 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 05:16:30.686881 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 05:16:30.686990 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 05:16:30.687045 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 05:16:30.688129 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0928 05:16:30.688208 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 05:16:30.692304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 23189
I20260928 05:16:30.693284 23251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:30.847266 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 23189
W20260928 05:16:30.868410 23354 connection.cc:535] server connection from 127.0.0.1:37706 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 05:16:30.869041 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 23263
I20260928 05:16:30.870200 23325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:30.980285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 23263
I20260928 05:16:31.000102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu pid 23337
I20260928 05:16:31.001370 23399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 05:16:31.106037 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzfOJKc/build/asan/bin/kudu with pid 23337
2026-09-28T05:16:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4949 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9260 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (362391 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST